2026-02-25T22:15:16.404 INFO:root:teuthology version: 1.2.4.dev11+g6e3d0419c 2026-02-25T22:15:16.418 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-25T22:15:16.494 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/emzhang-2026-02-25_22:10:53-rados:standalone-wip-mon-colocate-distro-default-trial/70701 branch: wip-mon-colocate description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} email: null first_in_suite: false flavor: default job_id: '70701' kernel: branch: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: trial name: emzhang-2026-02-25_22:10:53-rados:standalone-wip-mon-colocate-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: wip-mon-colocate ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd workunit: branch: wip-mon-colocate-test sha1: 61d4431d3ec879bd98659388b3655acf3d392418 owner: scheduled_emzhang@soko04.front.sepia.ceph.com priority: 50 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1952 sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd sleep_before_teardown: 0 suite: rados:standalone suite_branch: wip-mon-colocate-test suite_path: /home/teuthworker/src/github.com_Ericmzhang_ceph_61d4431d3ec879bd98659388b3655acf3d392418/qa suite_relpath: qa suite_repo: https://github.com/Ericmzhang/ceph.git suite_sha1: 61d4431d3ec879bd98659388b3655acf3d392418 targets: trial188.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBGEYkcFOLrKaKRRsHWcUA1E12uIt9BLAIuHiv0Ba8zt0SeeX6dGCyL1JR8eYgz4yyIrTdouKw6ujd4y6xX9pe4= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - misc teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6e3d0419ccca0e75a03e08feb708946e198236f4 timestamp: 2026-02-25_22:10:53 tube: trial user: emzhang verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.904699 2026-02-25T22:15:16.494 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_Ericmzhang_ceph_61d4431d3ec879bd98659388b3655acf3d392418/qa; will attempt to use it 2026-02-25T22:15:16.494 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_Ericmzhang_ceph_61d4431d3ec879bd98659388b3655acf3d392418/qa/tasks 2026-02-25T22:15:16.494 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-25T22:15:16.496 INFO:teuthology.task.internal:Checking packages... 2026-02-25T22:15:16.496 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd' 2026-02-25T22:15:16.496 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-25T22:15:16.496 INFO:teuthology.packaging:ref: None 2026-02-25T22:15:16.496 INFO:teuthology.packaging:tag: None 2026-02-25T22:15:16.496 INFO:teuthology.packaging:branch: wip-mon-colocate 2026-02-25T22:15:16.496 INFO:teuthology.packaging:sha1: 0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T22:15:16.496 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=wip-mon-colocate 2026-02-25T22:15:16.614 INFO:teuthology.task.internal:Found packages for ceph version 20.3.0-5305-g0886691e-1jammy 2026-02-25T22:15:16.615 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-25T22:15:16.627 INFO:teuthology.task.internal:no buildpackages task found 2026-02-25T22:15:16.627 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-25T22:15:16.641 INFO:teuthology.task.internal:Saving configuration 2026-02-25T22:15:16.648 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-25T22:15:16.660 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-25T22:15:16.737 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial188.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/emzhang-2026-02-25_22:10:53-rados:standalone-wip-mon-colocate-distro-default-trial/70701', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-25 22:11:45.450526', 'locked_by': 'scheduled_emzhang@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICV1nAqcibt9Hp+YRo4S7wJs4CNRjtoiAVpHWewDy/+B'} 2026-02-25T22:15:16.737 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-25T22:15:16.748 INFO:teuthology.task.internal:roles: ubuntu@trial188.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-25T22:15:16.749 INFO:teuthology.run_tasks:Running task console_log... 2026-02-25T22:15:16.831 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x79b41c626ca0>, signals=[15]) 2026-02-25T22:15:16.831 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-25T22:15:16.843 INFO:teuthology.task.internal:Opening connections... 2026-02-25T22:15:16.843 DEBUG:teuthology.task.internal:connecting to ubuntu@trial188.front.sepia.ceph.com 2026-02-25T22:15:16.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T22:15:16.907 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-25T22:15:16.918 DEBUG:teuthology.orchestra.run.trial188:> uname -m 2026-02-25T22:15:16.986 INFO:teuthology.orchestra.run.trial188.stdout:x86_64 2026-02-25T22:15:16.986 DEBUG:teuthology.orchestra.run.trial188:> cat /etc/os-release 2026-02-25T22:15:17.031 INFO:teuthology.orchestra.run.trial188.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-25T22:15:17.031 INFO:teuthology.orchestra.run.trial188.stdout:NAME="Ubuntu" 2026-02-25T22:15:17.031 INFO:teuthology.orchestra.run.trial188.stdout:VERSION_ID="22.04" 2026-02-25T22:15:17.031 INFO:teuthology.orchestra.run.trial188.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-25T22:15:17.031 INFO:teuthology.orchestra.run.trial188.stdout:VERSION_CODENAME=jammy 2026-02-25T22:15:17.031 INFO:teuthology.orchestra.run.trial188.stdout:ID=ubuntu 2026-02-25T22:15:17.031 INFO:teuthology.orchestra.run.trial188.stdout:ID_LIKE=debian 2026-02-25T22:15:17.032 INFO:teuthology.orchestra.run.trial188.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-25T22:15:17.032 INFO:teuthology.orchestra.run.trial188.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-25T22:15:17.032 INFO:teuthology.orchestra.run.trial188.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-25T22:15:17.032 INFO:teuthology.orchestra.run.trial188.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-25T22:15:17.032 INFO:teuthology.orchestra.run.trial188.stdout:UBUNTU_CODENAME=jammy 2026-02-25T22:15:17.032 INFO:teuthology.lock.ops:Updating trial188.front.sepia.ceph.com on lock server 2026-02-25T22:15:17.119 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-25T22:15:17.131 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-25T22:15:17.156 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-25T22:15:17.156 DEBUG:teuthology.orchestra.run.trial188:> test '!' -e /home/ubuntu/cephtest 2026-02-25T22:15:17.159 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-25T22:15:17.170 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-25T22:15:17.170 DEBUG:teuthology.orchestra.run.trial188:> test -z $(ls -A /var/lib/ceph) 2026-02-25T22:15:17.203 INFO:teuthology.orchestra.run.trial188.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-25T22:15:17.204 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-25T22:15:17.306 INFO:teuthology.run_tasks:Running task kernel... 2026-02-25T22:15:17.326 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'} 2026-02-25T22:15:17.326 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': 1, 'sha1': 'distro'}}, timeout 300 2026-02-25T22:15:17.326 DEBUG:teuthology.orchestra.run.trial188:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-25T22:15:17.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T22:15:17.330 DEBUG:teuthology.orchestra.run.trial188:> uname -r 2026-02-25T22:15:17.375 INFO:teuthology.orchestra.run.trial188.stdout:5.15.0-170-generic 2026-02-25T22:15:17.375 INFO:teuthology.task.kernel:Running kernel on trial188: 5.15.0-170-generic 2026-02-25T22:15:17.375 DEBUG:teuthology.orchestra.run.trial188:> sudo apt-get clean 2026-02-25T22:15:17.433 DEBUG:teuthology.orchestra.run.trial188:> sudo apt-get update 2026-02-25T22:15:17.982 INFO:teuthology.orchestra.run.trial188.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-25T22:15:18.066 INFO:teuthology.orchestra.run.trial188.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-25T22:15:18.383 INFO:teuthology.orchestra.run.trial188.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-25T22:15:18.523 INFO:teuthology.orchestra.run.trial188.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-25T22:15:18.654 INFO:teuthology.orchestra.run.trial188.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3274 kB] 2026-02-25T22:15:19.032 INFO:teuthology.orchestra.run.trial188.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [498 kB] 2026-02-25T22:15:19.043 INFO:teuthology.orchestra.run.trial188.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.2 kB] 2026-02-25T22:15:19.043 INFO:teuthology.orchestra.run.trial188.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-25T22:15:19.075 INFO:teuthology.orchestra.run.trial188.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-25T22:15:19.082 INFO:teuthology.orchestra.run.trial188.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [30.3 kB] 2026-02-25T22:15:19.083 INFO:teuthology.orchestra.run.trial188.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5400 kB] 2026-02-25T22:15:19.231 INFO:teuthology.orchestra.run.trial188.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1025 kB] 2026-02-25T22:15:19.253 INFO:teuthology.orchestra.run.trial188.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [3007 kB] 2026-02-25T22:15:19.338 INFO:teuthology.orchestra.run.trial188.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/main Translation-en [430 kB] 2026-02-25T22:15:19.344 INFO:teuthology.orchestra.run.trial188.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-25T22:15:19.344 INFO:teuthology.orchestra.run.trial188.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1019 kB] 2026-02-25T22:15:19.382 INFO:teuthology.orchestra.run.trial188.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy-security/universe Translation-en [225 kB] 2026-02-25T22:15:19.388 INFO:teuthology.orchestra.run.trial188.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [22.6 kB] 2026-02-25T22:15:19.389 INFO:teuthology.orchestra.run.trial188.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [5217 kB] 2026-02-25T22:15:19.523 INFO:teuthology.orchestra.run.trial188.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy-security/restricted Translation-en [993 kB] 2026-02-25T22:15:22.986 INFO:teuthology.orchestra.run.trial188.stdout:Fetched 23.1 MB in 3s (7970 kB/s) 2026-02-25T22:15:23.518 INFO:teuthology.orchestra.run.trial188.stdout:Reading package lists... 2026-02-25T22:15:23.532 DEBUG:teuthology.orchestra.run.trial188:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-25T22:15:23.565 INFO:teuthology.orchestra.run.trial188.stdout:Reading package lists... 2026-02-25T22:15:23.672 INFO:teuthology.orchestra.run.trial188.stdout:Building dependency tree... 2026-02-25T22:15:23.673 INFO:teuthology.orchestra.run.trial188.stdout:Reading state information... 2026-02-25T22:15:23.758 INFO:teuthology.orchestra.run.trial188.stdout:The following packages were automatically installed and are no longer required: 2026-02-25T22:15:23.758 INFO:teuthology.orchestra.run.trial188.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-25T22:15:23.758 INFO:teuthology.orchestra.run.trial188.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-25T22:15:23.758 INFO:teuthology.orchestra.run.trial188.stdout: sg3-utils-udev 2026-02-25T22:15:23.759 INFO:teuthology.orchestra.run.trial188.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-25T22:15:23.759 INFO:teuthology.orchestra.run.trial188.stdout:The following additional packages will be installed: 2026-02-25T22:15:23.759 INFO:teuthology.orchestra.run.trial188.stdout: linux-generic linux-headers-5.15.0-171 linux-headers-5.15.0-171-generic 2026-02-25T22:15:23.759 INFO:teuthology.orchestra.run.trial188.stdout: linux-headers-generic linux-image-5.15.0-171-generic 2026-02-25T22:15:23.759 INFO:teuthology.orchestra.run.trial188.stdout: linux-modules-5.15.0-171-generic linux-modules-extra-5.15.0-171-generic 2026-02-25T22:15:23.759 INFO:teuthology.orchestra.run.trial188.stdout:Suggested packages: 2026-02-25T22:15:23.759 INFO:teuthology.orchestra.run.trial188.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2026-02-25T22:15:23.771 INFO:teuthology.orchestra.run.trial188.stdout:The following NEW packages will be installed: 2026-02-25T22:15:23.771 INFO:teuthology.orchestra.run.trial188.stdout: linux-headers-5.15.0-171 linux-headers-5.15.0-171-generic 2026-02-25T22:15:23.771 INFO:teuthology.orchestra.run.trial188.stdout: linux-image-5.15.0-171-generic linux-modules-5.15.0-171-generic 2026-02-25T22:15:23.772 INFO:teuthology.orchestra.run.trial188.stdout: linux-modules-extra-5.15.0-171-generic 2026-02-25T22:15:23.772 INFO:teuthology.orchestra.run.trial188.stdout:The following packages will be upgraded: 2026-02-25T22:15:23.772 INFO:teuthology.orchestra.run.trial188.stdout: linux-generic linux-headers-generic linux-image-generic 2026-02-25T22:15:23.871 INFO:teuthology.orchestra.run.trial188.stdout:3 upgraded, 5 newly installed, 0 to remove and 31 not upgraded. 2026-02-25T22:15:23.871 INFO:teuthology.orchestra.run.trial188.stdout:Need to get 113 MB of archives. 2026-02-25T22:15:23.871 INFO:teuthology.orchestra.run.trial188.stdout:After this operation, 585 MB of additional disk space will be used. 2026-02-25T22:15:23.871 INFO:teuthology.orchestra.run.trial188.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-171-generic amd64 5.15.0-171.181 [22.7 MB] 2026-02-25T22:15:24.065 INFO:teuthology.orchestra.run.trial188.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-171-generic amd64 5.15.0-171.181 [11.6 MB] 2026-02-25T22:15:24.108 INFO:teuthology.orchestra.run.trial188.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-171-generic amd64 5.15.0-171.181 [63.9 MB] 2026-02-25T22:15:24.365 INFO:teuthology.orchestra.run.trial188.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.171.160 [1658 B] 2026-02-25T22:15:24.365 INFO:teuthology.orchestra.run.trial188.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.171.160 [2486 B] 2026-02-25T22:15:24.365 INFO:teuthology.orchestra.run.trial188.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-171 all 5.15.0-171.181 [12.4 MB] 2026-02-25T22:15:24.416 INFO:teuthology.orchestra.run.trial188.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-171-generic amd64 5.15.0-171.181 [2849 kB] 2026-02-25T22:15:24.427 INFO:teuthology.orchestra.run.trial188.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.171.160 [2332 B] 2026-02-25T22:15:24.626 INFO:teuthology.orchestra.run.trial188.stdout:Fetched 113 MB in 1s (174 MB/s) 2026-02-25T22:15:24.646 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package linux-modules-5.15.0-171-generic. 2026-02-25T22:15:24.817 INFO:teuthology.orchestra.run.trial188.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167442 files and directories currently installed.) 2026-02-25T22:15:24.819 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../0-linux-modules-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T22:15:24.820 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking linux-modules-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:25.162 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package linux-image-5.15.0-171-generic. 2026-02-25T22:15:25.169 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../1-linux-image-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T22:15:25.189 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:25.232 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-171-generic. 2026-02-25T22:15:25.240 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T22:15:25.241 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking linux-modules-extra-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:26.362 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../3-linux-generic_5.15.0.171.160_amd64.deb ... 2026-02-25T22:15:26.363 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking linux-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-25T22:15:26.388 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.171.160_amd64.deb ... 2026-02-25T22:15:26.389 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking linux-image-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-25T22:15:26.405 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package linux-headers-5.15.0-171. 2026-02-25T22:15:26.414 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../5-linux-headers-5.15.0-171_5.15.0-171.181_all.deb ... 2026-02-25T22:15:26.414 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking linux-headers-5.15.0-171 (5.15.0-171.181) ... 2026-02-25T22:15:27.774 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package linux-headers-5.15.0-171-generic. 2026-02-25T22:15:27.788 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../6-linux-headers-5.15.0-171-generic_5.15.0-171.181_amd64.deb ... 2026-02-25T22:15:27.789 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking linux-headers-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:28.387 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.171.160_amd64.deb ... 2026-02-25T22:15:28.388 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking linux-headers-generic (5.15.0.171.160) over (5.15.0.170.159) ... 2026-02-25T22:15:28.414 INFO:teuthology.orchestra.run.trial188.stdout:Setting up linux-headers-5.15.0-171 (5.15.0-171.181) ... 2026-02-25T22:15:28.415 INFO:teuthology.orchestra.run.trial188.stdout:Setting up linux-headers-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:28.417 INFO:teuthology.orchestra.run.trial188.stdout:Setting up linux-headers-generic (5.15.0.171.160) ... 2026-02-25T22:15:28.419 INFO:teuthology.orchestra.run.trial188.stdout:Setting up linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:29.207 INFO:teuthology.orchestra.run.trial188.stdout:I: /boot/vmlinuz.old is now a symlink to vmlinuz-5.15.0-170-generic 2026-02-25T22:15:29.207 INFO:teuthology.orchestra.run.trial188.stdout:I: /boot/initrd.img.old is now a symlink to initrd.img-5.15.0-170-generic 2026-02-25T22:15:29.208 INFO:teuthology.orchestra.run.trial188.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-171-generic 2026-02-25T22:15:29.208 INFO:teuthology.orchestra.run.trial188.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-171-generic 2026-02-25T22:15:29.215 INFO:teuthology.orchestra.run.trial188.stdout:Setting up linux-modules-extra-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:30.019 INFO:teuthology.orchestra.run.trial188.stdout:Setting up linux-modules-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:30.838 INFO:teuthology.orchestra.run.trial188.stdout:Setting up linux-image-generic (5.15.0.171.160) ... 2026-02-25T22:15:30.839 INFO:teuthology.orchestra.run.trial188.stdout:Setting up linux-generic (5.15.0.171.160) ... 2026-02-25T22:15:30.840 INFO:teuthology.orchestra.run.trial188.stdout:Processing triggers for linux-image-5.15.0-171-generic (5.15.0-171.181) ... 2026-02-25T22:15:30.846 INFO:teuthology.orchestra.run.trial188.stdout:/etc/kernel/postinst.d/initramfs-tools: 2026-02-25T22:15:30.846 INFO:teuthology.orchestra.run.trial188.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-171-generic 2026-02-25T22:15:37.936 INFO:teuthology.orchestra.run.trial188.stdout:W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast 2026-02-25T22:15:44.575 INFO:teuthology.orchestra.run.trial188.stdout:/etc/kernel/postinst.d/zz-update-grub: 2026-02-25T22:15:44.576 INFO:teuthology.orchestra.run.trial188.stdout:Sourcing file `/etc/default/grub' 2026-02-25T22:15:44.595 INFO:teuthology.orchestra.run.trial188.stdout:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-25T22:15:44.596 INFO:teuthology.orchestra.run.trial188.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-25T22:15:44.598 INFO:teuthology.orchestra.run.trial188.stdout:Generating grub configuration file ... 2026-02-25T22:15:44.684 INFO:teuthology.orchestra.run.trial188.stdout:Found linux image: /boot/vmlinuz-5.15.0-171-generic 2026-02-25T22:15:44.691 INFO:teuthology.orchestra.run.trial188.stdout:Found initrd image: /boot/initrd.img-5.15.0-171-generic 2026-02-25T22:15:44.873 INFO:teuthology.orchestra.run.trial188.stdout:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-25T22:15:44.879 INFO:teuthology.orchestra.run.trial188.stdout:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-25T22:15:44.923 INFO:teuthology.orchestra.run.trial188.stdout:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-25T22:15:44.929 INFO:teuthology.orchestra.run.trial188.stdout:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-25T22:15:45.006 INFO:teuthology.orchestra.run.trial188.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-25T22:15:45.006 INFO:teuthology.orchestra.run.trial188.stdout:Systems on them will not be added to the GRUB boot configuration. 2026-02-25T22:15:45.006 INFO:teuthology.orchestra.run.trial188.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-25T22:15:45.009 INFO:teuthology.orchestra.run.trial188.stdout:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T22:15:45.018 INFO:teuthology.orchestra.run.trial188.stdout:done 2026-02-25T22:15:45.434 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:15:45.434 INFO:teuthology.orchestra.run.trial188.stdout:Pending kernel upgrade! 2026-02-25T22:15:45.434 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:15:45.434 INFO:teuthology.orchestra.run.trial188.stdout:Running kernel version: 2026-02-25T22:15:45.434 INFO:teuthology.orchestra.run.trial188.stdout: 5.15.0-170-generic 2026-02-25T22:15:45.434 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:15:45.434 INFO:teuthology.orchestra.run.trial188.stdout:Diagnostics: 2026-02-25T22:15:45.434 INFO:teuthology.orchestra.run.trial188.stdout: The currently running kernel version is not the expected kernel version 5.15.0-171-generic. 2026-02-25T22:15:45.434 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:15:45.434 INFO:teuthology.orchestra.run.trial188.stdout:Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting. [Return] 2026-02-25T22:15:45.434 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:15:45.435 INFO:teuthology.orchestra.run.trial188.stdout:The processor microcode seems to be up-to-date. 2026-02-25T22:15:45.435 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:15:45.435 INFO:teuthology.orchestra.run.trial188.stdout:No services need to be restarted. 2026-02-25T22:15:45.435 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:15:45.435 INFO:teuthology.orchestra.run.trial188.stdout:No containers need to be restarted. 2026-02-25T22:15:45.435 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:15:45.435 INFO:teuthology.orchestra.run.trial188.stdout:No user sessions are running outdated binaries. 2026-02-25T22:15:45.435 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:15:45.435 INFO:teuthology.orchestra.run.trial188.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-25T22:15:46.257 DEBUG:teuthology.orchestra.run.trial188:> dpkg -s linux-image-generic 2026-02-25T22:15:46.310 INFO:teuthology.orchestra.run.trial188.stdout:Package: linux-image-generic 2026-02-25T22:15:46.310 INFO:teuthology.orchestra.run.trial188.stdout:Status: install ok installed 2026-02-25T22:15:46.310 INFO:teuthology.orchestra.run.trial188.stdout:Priority: optional 2026-02-25T22:15:46.310 INFO:teuthology.orchestra.run.trial188.stdout:Section: kernel 2026-02-25T22:15:46.310 INFO:teuthology.orchestra.run.trial188.stdout:Installed-Size: 22 2026-02-25T22:15:46.310 INFO:teuthology.orchestra.run.trial188.stdout:Maintainer: Ubuntu Kernel Team 2026-02-25T22:15:46.310 INFO:teuthology.orchestra.run.trial188.stdout:Architecture: amd64 2026-02-25T22:15:46.310 INFO:teuthology.orchestra.run.trial188.stdout:Source: linux-meta 2026-02-25T22:15:46.311 INFO:teuthology.orchestra.run.trial188.stdout:Version: 5.15.0.171.160 2026-02-25T22:15:46.311 INFO:teuthology.orchestra.run.trial188.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-171), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-25T22:15:46.311 INFO:teuthology.orchestra.run.trial188.stdout:Depends: linux-image-5.15.0-171-generic, linux-modules-extra-5.15.0-171-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-25T22:15:46.311 INFO:teuthology.orchestra.run.trial188.stdout:Recommends: thermald 2026-02-25T22:15:46.311 INFO:teuthology.orchestra.run.trial188.stdout:Description: Generic Linux kernel image 2026-02-25T22:15:46.311 INFO:teuthology.orchestra.run.trial188.stdout: This package will always depend on the latest generic kernel image 2026-02-25T22:15:46.311 INFO:teuthology.orchestra.run.trial188.stdout: available. 2026-02-25T22:15:46.311 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-170-generic Expected: 5.15.0-171-generic 2026-02-25T22:15:46.312 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-25T22:15:46.312 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-25T22:15:46.312 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-25T22:15:46.312 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial188.front.sepia.ceph.com, path=None, version=distro) 2026-02-25T22:15:46.312 DEBUG:teuthology.orchestra.run.trial188:> sudo apt-get clean 2026-02-25T22:15:46.392 DEBUG:teuthology.orchestra.run.trial188:> sudo apt-get update 2026-02-25T22:15:46.883 INFO:teuthology.orchestra.run.trial188.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-25T22:15:46.966 INFO:teuthology.orchestra.run.trial188.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-25T22:15:47.055 INFO:teuthology.orchestra.run.trial188.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-25T22:15:47.143 INFO:teuthology.orchestra.run.trial188.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-25T22:15:47.827 INFO:teuthology.orchestra.run.trial188.stdout:Reading package lists... 2026-02-25T22:15:47.842 DEBUG:teuthology.orchestra.run.trial188:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-25T22:15:47.874 INFO:teuthology.orchestra.run.trial188.stdout:Reading package lists... 2026-02-25T22:15:47.986 INFO:teuthology.orchestra.run.trial188.stdout:Building dependency tree... 2026-02-25T22:15:47.986 INFO:teuthology.orchestra.run.trial188.stdout:Reading state information... 2026-02-25T22:15:48.077 INFO:teuthology.orchestra.run.trial188.stdout:linux-image-generic is already the newest version (5.15.0.171.160). 2026-02-25T22:15:48.077 INFO:teuthology.orchestra.run.trial188.stdout:linux-image-generic set to manually installed. 2026-02-25T22:15:48.077 INFO:teuthology.orchestra.run.trial188.stdout:The following packages were automatically installed and are no longer required: 2026-02-25T22:15:48.077 INFO:teuthology.orchestra.run.trial188.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-25T22:15:48.077 INFO:teuthology.orchestra.run.trial188.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 2026-02-25T22:15:48.077 INFO:teuthology.orchestra.run.trial188.stdout: linux-headers-5.15.0-164 linux-headers-5.15.0-164-generic 2026-02-25T22:15:48.077 INFO:teuthology.orchestra.run.trial188.stdout: linux-image-5.15.0-164-generic linux-modules-5.15.0-164-generic 2026-02-25T22:15:48.077 INFO:teuthology.orchestra.run.trial188.stdout: linux-modules-extra-5.15.0-164-generic sg3-utils sg3-utils-udev 2026-02-25T22:15:48.077 INFO:teuthology.orchestra.run.trial188.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-25T22:15:48.097 INFO:teuthology.orchestra.run.trial188.stdout:0 upgraded, 0 newly installed, 0 to remove and 31 not upgraded. 2026-02-25T22:15:48.100 DEBUG:teuthology.orchestra.run.trial188:> dpkg -s linux-image-generic 2026-02-25T22:15:48.149 INFO:teuthology.orchestra.run.trial188.stdout:Package: linux-image-generic 2026-02-25T22:15:48.149 INFO:teuthology.orchestra.run.trial188.stdout:Status: install ok installed 2026-02-25T22:15:48.149 INFO:teuthology.orchestra.run.trial188.stdout:Priority: optional 2026-02-25T22:15:48.149 INFO:teuthology.orchestra.run.trial188.stdout:Section: kernel 2026-02-25T22:15:48.149 INFO:teuthology.orchestra.run.trial188.stdout:Installed-Size: 22 2026-02-25T22:15:48.149 INFO:teuthology.orchestra.run.trial188.stdout:Maintainer: Ubuntu Kernel Team 2026-02-25T22:15:48.149 INFO:teuthology.orchestra.run.trial188.stdout:Architecture: amd64 2026-02-25T22:15:48.150 INFO:teuthology.orchestra.run.trial188.stdout:Source: linux-meta 2026-02-25T22:15:48.150 INFO:teuthology.orchestra.run.trial188.stdout:Version: 5.15.0.171.160 2026-02-25T22:15:48.150 INFO:teuthology.orchestra.run.trial188.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-171), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-25T22:15:48.150 INFO:teuthology.orchestra.run.trial188.stdout:Depends: linux-image-5.15.0-171-generic, linux-modules-extra-5.15.0-171-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-25T22:15:48.150 INFO:teuthology.orchestra.run.trial188.stdout:Recommends: thermald 2026-02-25T22:15:48.150 INFO:teuthology.orchestra.run.trial188.stdout:Description: Generic Linux kernel image 2026-02-25T22:15:48.150 INFO:teuthology.orchestra.run.trial188.stdout: This package will always depend on the latest generic kernel image 2026-02-25T22:15:48.150 INFO:teuthology.orchestra.run.trial188.stdout: available. 2026-02-25T22:15:48.151 DEBUG:teuthology.orchestra.run.trial188:> mktemp 2026-02-25T22:15:48.196 INFO:teuthology.orchestra.run.trial188.stdout:/tmp/tmp.BslKInUCXS 2026-02-25T22:15:48.197 DEBUG:teuthology.orchestra.run.trial188:> sudo cp /boot/grub/grub.cfg /tmp/tmp.BslKInUCXS 2026-02-25T22:15:48.249 DEBUG:teuthology.orchestra.run.trial188:> sudo chmod 0666 /tmp/tmp.BslKInUCXS 2026-02-25T22:15:48.346 DEBUG:teuthology.orchestra.remote:trial188:/tmp/tmp.BslKInUCXS is 9KB 2026-02-25T22:15:48.354 DEBUG:teuthology.orchestra.run.trial188:> rm -fr /tmp/tmp.BslKInUCXS 2026-02-25T22:15:48.359 DEBUG:teuthology.orchestra.run.trial188:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2026-02-25T22:15:48.408 DEBUG:teuthology.orchestra.run.trial188:> set -ex 2026-02-25T22:15:48.408 DEBUG:teuthology.orchestra.run.trial188:> sudo dd of=/etc/grub.d/01_ceph_kernel 2026-02-25T22:15:48.408 DEBUG:teuthology.orchestra.run.trial188:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2026-02-25T22:15:48.462 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-171-generic 2026-02-25T22:15:48.462 DEBUG:teuthology.orchestra.run.trial188:> sudo update-grub 2026-02-25T22:15:48.700 INFO:teuthology.orchestra.run.trial188.stderr:Sourcing file `/etc/default/grub' 2026-02-25T22:15:48.721 INFO:teuthology.orchestra.run.trial188.stderr:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-25T22:15:48.722 INFO:teuthology.orchestra.run.trial188.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-25T22:15:48.724 INFO:teuthology.orchestra.run.trial188.stderr:Generating grub configuration file ... 2026-02-25T22:15:48.807 INFO:teuthology.orchestra.run.trial188.stderr:Found linux image: /boot/vmlinuz-5.15.0-171-generic 2026-02-25T22:15:48.813 INFO:teuthology.orchestra.run.trial188.stderr:Found initrd image: /boot/initrd.img-5.15.0-171-generic 2026-02-25T22:15:48.987 INFO:teuthology.orchestra.run.trial188.stderr:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-25T22:15:48.992 INFO:teuthology.orchestra.run.trial188.stderr:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-25T22:15:49.033 INFO:teuthology.orchestra.run.trial188.stderr:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-25T22:15:49.039 INFO:teuthology.orchestra.run.trial188.stderr:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-25T22:15:49.111 INFO:teuthology.orchestra.run.trial188.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-25T22:15:49.111 INFO:teuthology.orchestra.run.trial188.stderr:Systems on them will not be added to the GRUB boot configuration. 2026-02-25T22:15:49.111 INFO:teuthology.orchestra.run.trial188.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-25T22:15:49.113 INFO:teuthology.orchestra.run.trial188.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-25T22:15:49.121 INFO:teuthology.orchestra.run.trial188.stderr:done 2026-02-25T22:15:49.123 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial188 2026-02-25T22:15:49.123 DEBUG:teuthology.orchestra.run.trial188:> sudo bash -c ' 2026-02-25T22:15:49.123 DEBUG:teuthology.orchestra.run.trial188:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-25T22:15:49.123 DEBUG:teuthology.orchestra.run.trial188:> ' 2026-02-25T22:16:19.125 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-25T22:16:19.126 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial188.front.sepia.ceph.com' 2026-02-25T22:16:19.127 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T22:16:33.408 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 10.20.193.188 2026-02-25T22:16:42.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial188.front.sepia.ceph.com' 2026-02-25T22:16:42.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T22:17:42.448 DEBUG:teuthology.orchestra.remote:timed out 2026-02-25T22:17:54.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial188.front.sepia.ceph.com' 2026-02-25T22:17:54.461 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T22:17:54.588 DEBUG:teuthology.orchestra.run.trial188:> true 2026-02-25T22:17:54.979 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial188.front.sepia.ceph.com' 2026-02-25T22:17:54.980 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-171-generic"... 2026-02-25T22:17:54.980 DEBUG:teuthology.orchestra.run.trial188:> uname -r 2026-02-25T22:17:55.023 INFO:teuthology.orchestra.run.trial188.stdout:5.15.0-171-generic 2026-02-25T22:17:55.024 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-171-generic vs 5.15.0-171-generic 2026-02-25T22:17:55.024 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-25T22:17:55.024 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2026-02-25T22:17:56.025 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-25T22:17:56.025 DEBUG:teuthology.orchestra.run.trial188:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-25T22:17:56.045 INFO:teuthology.orchestra.run.trial188.stdout:ttyS1 2026-02-25T22:17:56.074 DEBUG:teuthology.parallel:result is None 2026-02-25T22:17:56.075 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-25T22:17:56.086 INFO:teuthology.task.internal:Creating test directory... 2026-02-25T22:17:56.086 DEBUG:teuthology.orchestra.run.trial188:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-25T22:17:56.120 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-25T22:17:56.132 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-25T22:17:56.146 INFO:teuthology.task.internal:Creating archive directory... 2026-02-25T22:17:56.147 DEBUG:teuthology.orchestra.run.trial188:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-25T22:17:56.170 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-25T22:17:56.182 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-25T22:17:56.182 DEBUG:teuthology.orchestra.run.trial188:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-25T22:17:56.218 INFO:teuthology.orchestra.run.trial188.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:17:56.224 INFO:teuthology.orchestra.run.trial188.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:17:56.226 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-25T22:17:56.238 INFO:teuthology.task.internal:Configuring sudo... 2026-02-25T22:17:56.238 DEBUG:teuthology.orchestra.run.trial188:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-25T22:17:56.277 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-25T22:17:56.293 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-25T22:17:56.294 DEBUG:teuthology.orchestra.run.trial188:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-25T22:17:56.324 DEBUG:teuthology.orchestra.run.trial188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-25T22:17:56.368 DEBUG:teuthology.orchestra.run.trial188:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-25T22:17:56.416 DEBUG:teuthology.orchestra.run.trial188:> set -ex 2026-02-25T22:17:56.416 DEBUG:teuthology.orchestra.run.trial188:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-25T22:17:56.466 DEBUG:teuthology.orchestra.run.trial188:> sudo service rsyslog restart 2026-02-25T22:17:56.525 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-25T22:17:56.537 INFO:teuthology.task.internal:Starting timer... 2026-02-25T22:17:56.538 INFO:teuthology.run_tasks:Running task pcp... 2026-02-25T22:17:56.560 INFO:teuthology.run_tasks:Running task selinux... 2026-02-25T22:17:56.577 DEBUG:teuthology.task.selinux:Excluding trial188: OS 'ubuntu' does not support SELinux 2026-02-25T22:17:56.577 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-25T22:17:56.578 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-25T22:17:56.578 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-25T22:17:56.578 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-25T22:17:56.593 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-25T22:17:56.607 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-25T22:17:56.608 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial188.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-25T22:20:25.315 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial188.front.sepia.ceph.com')] 2026-02-25T22:20:25.317 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial188.front.sepia.ceph.com' 2026-02-25T22:20:25.318 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial188.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-25T22:20:25.426 DEBUG:teuthology.orchestra.run.trial188:> true 2026-02-25T22:20:25.670 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial188.front.sepia.ceph.com' 2026-02-25T22:20:25.670 INFO:teuthology.run_tasks:Running task clock... 2026-02-25T22:20:25.687 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-25T22:20:25.687 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-25T22:20:25.687 DEBUG:teuthology.orchestra.run.trial188:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-25T22:20:25.730 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-25T22:20:25.730 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: Command line: ntpd -gq 2026-02-25T22:20:25.730 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: ---------------------------------------------------- 2026-02-25T22:20:25.730 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: ntp-4 is maintained by Network Time Foundation, 2026-02-25T22:20:25.730 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-25T22:20:25.730 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: corporation. Support and training for ntp-4 are 2026-02-25T22:20:25.731 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: available at https://www.nwtime.org/support 2026-02-25T22:20:25.731 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: ---------------------------------------------------- 2026-02-25T22:20:25.731 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: proto: precision = 0.020 usec (-25) 2026-02-25T22:20:25.731 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: basedate set to 2022-02-04 2026-02-25T22:20:25.731 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: gps base set to 2022-02-06 (week 2196) 2026-02-25T22:20:25.731 INFO:teuthology.orchestra.run.trial188.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-25T22:20:25.732 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-25T22:20:25.732 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: restrict ::: KOD does nothing without LIMITED. 2026-02-25T22:20:25.732 INFO:teuthology.orchestra.run.trial188.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-25T22:20:25.733 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: Listen and drop on 0 v6wildcard [::]:123 2026-02-25T22:20:25.733 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-25T22:20:25.733 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: Listen normally on 2 lo 127.0.0.1:123 2026-02-25T22:20:25.733 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: Listen normally on 3 enp1s0f0 10.20.193.188:123 2026-02-25T22:20:25.733 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: Listen normally on 4 lo [::1]:123 2026-02-25T22:20:25.734 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:5fac%2]:123 2026-02-25T22:20:25.734 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:25 ntpd[8819]: Listening on routing socket on fd #22 for interface updates 2026-02-25T22:20:26.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:26.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:26.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:26.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:26.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:26.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:26.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:26.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:26.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:20:26.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:20:26.742 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:26.742 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:26.742 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:20:26.743 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:20:26.752 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:26.752 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:26.752 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:20:26.753 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:26 ntpd[8819]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:20:28.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:28.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:28.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:28.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:28.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:28.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:28.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:28.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:28.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:20:28.734 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:20:28.742 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:28.742 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:28.742 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:20:28.742 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:20:28.748 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:28.748 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:28.749 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:20:28.749 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:28 ntpd[8819]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:20:29.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:29 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:29.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:29 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:29.806 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:29 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:29.806 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:29 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:29.807 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:29 ntpd[8819]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:20:29.807 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:29 ntpd[8819]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:20:30.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:30.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:30.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:30.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:30.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:30.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:30.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:30.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:30.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:20:30.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:20:30.742 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:30.742 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:30.742 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:20:30.742 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:20:30.753 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:30.753 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:30.753 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:20:30.753 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:30 ntpd[8819]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:20:31.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:31 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:31.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:31 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:31.801 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:31 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:31.801 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:31 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:31.801 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:31 ntpd[8819]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:20:31.801 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:31 ntpd[8819]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:20:32.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:32 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:32.730 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:32 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:32.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:32 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:32.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:32 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:32.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:32 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:32.731 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:32 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:32.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:32 ntpd[8819]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-25T22:20:32.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:32 ntpd[8819]: can't open /var/log/ntpstats/rawstats.20260225: Permission denied 2026-02-25T22:20:32.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:32 ntpd[8819]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-25T22:20:32.733 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:32 ntpd[8819]: can't open /var/log/ntpstats/peerstats.20260225: Permission denied 2026-02-25T22:20:32.734 INFO:teuthology.orchestra.run.trial188.stdout:25 Feb 22:20:32 ntpd[8819]: ntpd: time set +0.070828 s 2026-02-25T22:20:32.734 INFO:teuthology.orchestra.run.trial188.stdout:ntpd: time set +0.070828s 2026-02-25T22:20:32.734 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:32 ntpd[8819]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-25T22:20:32.734 INFO:teuthology.orchestra.run.trial188.stderr:25 Feb 22:20:32 ntpd[8819]: can't open /var/log/ntpstats/loopstats.20260225: Permission denied 2026-02-25T22:20:32.757 INFO:teuthology.orchestra.run.trial188.stdout:No association ID's returned 2026-02-25T22:20:32.758 INFO:teuthology.run_tasks:Running task install... 2026-02-25T22:20:32.775 DEBUG:teuthology.task.install:project ceph 2026-02-25T22:20:32.775 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd'}} 2026-02-25T22:20:32.776 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '0886691e8f7870d54dd61f71a1fe5f8750d0f7dd'} 2026-02-25T22:20:32.776 INFO:teuthology.task.install:Using flavor: default 2026-02-25T22:20:32.784 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-osd', 'ceph-osd-classic', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-osd', 'ceph-osd-classic', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-25T22:20:32.785 INFO:teuthology.task.install:extra packages: [] 2026-02-25T22:20:32.785 DEBUG:teuthology.orchestra.run.trial188:> sudo apt-key list | grep Ceph 2026-02-25T22:20:32.852 INFO:teuthology.orchestra.run.trial188.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-25T22:20:32.879 INFO:teuthology.orchestra.run.trial188.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-25T22:20:32.879 INFO:teuthology.orchestra.run.trial188.stdout:uid [ unknown] Ceph.com (release key) 2026-02-25T22:20:32.880 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-osd, ceph-osd-classic, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-25T22:20:32.881 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T22:20:33.056 INFO:teuthology.task.install.deb:Pulling from https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default/ 2026-02-25T22:20:33.056 INFO:teuthology.task.install.deb:Package version is 20.3.0-5305-g0886691e-1jammy 2026-02-25T22:20:33.299 DEBUG:teuthology.orchestra.run.trial188:> set -ex 2026-02-25T22:20:33.299 DEBUG:teuthology.orchestra.run.trial188:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-25T22:20:33.308 DEBUG:teuthology.orchestra.run.trial188:> sudo apt-get update 2026-02-25T22:20:33.555 INFO:teuthology.orchestra.run.trial188.stdout:Ign:1 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy InRelease 2026-02-25T22:20:33.589 INFO:teuthology.orchestra.run.trial188.stdout:Get:2 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy Release [7689 B] 2026-02-25T22:20:33.624 INFO:teuthology.orchestra.run.trial188.stdout:Ign:3 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-25T22:20:33.658 INFO:teuthology.orchestra.run.trial188.stdout:Get:4 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 Packages [19.2 kB] 2026-02-25T22:20:33.962 INFO:teuthology.orchestra.run.trial188.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-25T22:20:34.046 INFO:teuthology.orchestra.run.trial188.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-25T22:20:34.134 INFO:teuthology.orchestra.run.trial188.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-25T22:20:34.223 INFO:teuthology.orchestra.run.trial188.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-25T22:20:34.366 INFO:teuthology.orchestra.run.trial188.stdout:Fetched 26.9 kB in 1s (30.0 kB/s) 2026-02-25T22:20:34.907 INFO:teuthology.orchestra.run.trial188.stdout:Reading package lists... 2026-02-25T22:20:34.921 DEBUG:teuthology.orchestra.run.trial188:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=20.3.0-5305-g0886691e-1jammy cephadm=20.3.0-5305-g0886691e-1jammy ceph-mds=20.3.0-5305-g0886691e-1jammy ceph-mgr=20.3.0-5305-g0886691e-1jammy ceph-osd=20.3.0-5305-g0886691e-1jammy ceph-osd-classic=20.3.0-5305-g0886691e-1jammy ceph-common=20.3.0-5305-g0886691e-1jammy ceph-fuse=20.3.0-5305-g0886691e-1jammy ceph-test=20.3.0-5305-g0886691e-1jammy ceph-volume=20.3.0-5305-g0886691e-1jammy radosgw=20.3.0-5305-g0886691e-1jammy python3-rados=20.3.0-5305-g0886691e-1jammy python3-rgw=20.3.0-5305-g0886691e-1jammy python3-cephfs=20.3.0-5305-g0886691e-1jammy python3-rbd=20.3.0-5305-g0886691e-1jammy libcephfs2=20.3.0-5305-g0886691e-1jammy libcephfs-dev=20.3.0-5305-g0886691e-1jammy librados2=20.3.0-5305-g0886691e-1jammy librbd1=20.3.0-5305-g0886691e-1jammy rbd-fuse=20.3.0-5305-g0886691e-1jammy 2026-02-25T22:20:34.956 INFO:teuthology.orchestra.run.trial188.stdout:Reading package lists... 2026-02-25T22:20:35.066 INFO:teuthology.orchestra.run.trial188.stdout:Building dependency tree... 2026-02-25T22:20:35.066 INFO:teuthology.orchestra.run.trial188.stdout:Reading state information... 2026-02-25T22:20:35.156 INFO:teuthology.orchestra.run.trial188.stdout:The following packages were automatically installed and are no longer required: 2026-02-25T22:20:35.156 INFO:teuthology.orchestra.run.trial188.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-25T22:20:35.156 INFO:teuthology.orchestra.run.trial188.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 2026-02-25T22:20:35.156 INFO:teuthology.orchestra.run.trial188.stdout: linux-headers-5.15.0-164 linux-headers-5.15.0-164-generic 2026-02-25T22:20:35.156 INFO:teuthology.orchestra.run.trial188.stdout: linux-image-5.15.0-164-generic linux-modules-5.15.0-164-generic 2026-02-25T22:20:35.157 INFO:teuthology.orchestra.run.trial188.stdout: linux-modules-extra-5.15.0-164-generic sg3-utils sg3-utils-udev 2026-02-25T22:20:35.157 INFO:teuthology.orchestra.run.trial188.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-25T22:20:35.157 INFO:teuthology.orchestra.run.trial188.stdout:The following additional packages will be installed: 2026-02-25T22:20:35.157 INFO:teuthology.orchestra.run.trial188.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-25T22:20:35.157 INFO:teuthology.orchestra.run.trial188.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon jq libcephfs-daemon 2026-02-25T22:20:35.157 INFO:teuthology.orchestra.run.trial188.stdout: libcephfs-proxy2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 2026-02-25T22:20:35.157 INFO:teuthology.orchestra.run.trial188.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-25T22:20:35.157 INFO:teuthology.orchestra.run.trial188.stdout: libradosstriper1 librdkafka1 librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2026-02-25T22:20:35.157 INFO:teuthology.orchestra.run.trial188.stdout: nvme-cli python-asyncssh-doc python3-asyncssh python3-cachetools 2026-02-25T22:20:35.157 INFO:teuthology.orchestra.run.trial188.stdout: python3-ceph-argparse python3-ceph-common python3-cheroot python3-cherrypy3 2026-02-25T22:20:35.157 INFO:teuthology.orchestra.run.trial188.stdout: python3-google-auth python3-iniconfig python3-jaraco.classes 2026-02-25T22:20:35.157 INFO:teuthology.orchestra.run.trial188.stdout: python3-jaraco.collections python3-jaraco.functools python3-jaraco.text 2026-02-25T22:20:35.158 INFO:teuthology.orchestra.run.trial188.stdout: python3-joblib python3-kubernetes python3-natsort python3-pluggy 2026-02-25T22:20:35.158 INFO:teuthology.orchestra.run.trial188.stdout: python3-portend python3-prettytable python3-psutil python3-py 2026-02-25T22:20:35.158 INFO:teuthology.orchestra.run.trial188.stdout: python3-pygments python3-pytest python3-repoze.lru python3-requests-oauthlib 2026-02-25T22:20:35.158 INFO:teuthology.orchestra.run.trial188.stdout: python3-routes python3-rsa python3-simplejson python3-sklearn 2026-02-25T22:20:35.158 INFO:teuthology.orchestra.run.trial188.stdout: python3-sklearn-lib python3-tempora python3-threadpoolctl python3-toml 2026-02-25T22:20:35.158 INFO:teuthology.orchestra.run.trial188.stdout: python3-wcwidth python3-webob python3-websocket python3-zc.lockfile 2026-02-25T22:20:35.158 INFO:teuthology.orchestra.run.trial188.stdout: qttranslations5-l10n socat xmlstarlet 2026-02-25T22:20:35.158 INFO:teuthology.orchestra.run.trial188.stdout:Suggested packages: 2026-02-25T22:20:35.158 INFO:teuthology.orchestra.run.trial188.stdout: python3-influxdb liblua5.3-dev luarocks python-natsort-doc python-psutil-doc 2026-02-25T22:20:35.158 INFO:teuthology.orchestra.run.trial188.stdout: subversion python-pygments-doc ttf-bitstream-vera python3-paste python3-dap 2026-02-25T22:20:35.158 INFO:teuthology.orchestra.run.trial188.stdout: python-sklearn-doc ipython3 python-webob-doc 2026-02-25T22:20:35.158 INFO:teuthology.orchestra.run.trial188.stdout:Recommended packages: 2026-02-25T22:20:35.158 INFO:teuthology.orchestra.run.trial188.stdout: btrfs-tools 2026-02-25T22:20:35.191 INFO:teuthology.orchestra.run.trial188.stdout:The following NEW packages will be installed: 2026-02-25T22:20:35.191 INFO:teuthology.orchestra.run.trial188.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-25T22:20:35.191 INFO:teuthology.orchestra.run.trial188.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-25T22:20:35.191 INFO:teuthology.orchestra.run.trial188.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-osd-classic ceph-test 2026-02-25T22:20:35.192 INFO:teuthology.orchestra.run.trial188.stdout: ceph-volume cephadm jq libcephfs-daemon libcephfs-dev libcephfs-proxy2 2026-02-25T22:20:35.192 INFO:teuthology.orchestra.run.trial188.stdout: libcephfs2 libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 2026-02-25T22:20:35.192 INFO:teuthology.orchestra.run.trial188.stdout: liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 2026-02-25T22:20:35.192 INFO:teuthology.orchestra.run.trial188.stdout: librados2 libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2026-02-25T22:20:35.192 INFO:teuthology.orchestra.run.trial188.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python3-asyncssh 2026-02-25T22:20:35.192 INFO:teuthology.orchestra.run.trial188.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2026-02-25T22:20:35.192 INFO:teuthology.orchestra.run.trial188.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-25T22:20:35.192 INFO:teuthology.orchestra.run.trial188.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-25T22:20:35.192 INFO:teuthology.orchestra.run.trial188.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-natsort 2026-02-25T22:20:35.192 INFO:teuthology.orchestra.run.trial188.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2026-02-25T22:20:35.192 INFO:teuthology.orchestra.run.trial188.stdout: python3-pygments python3-pytest python3-rados python3-rbd python3-repoze.lru 2026-02-25T22:20:35.192 INFO:teuthology.orchestra.run.trial188.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2026-02-25T22:20:35.192 INFO:teuthology.orchestra.run.trial188.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2026-02-25T22:20:35.193 INFO:teuthology.orchestra.run.trial188.stdout: python3-threadpoolctl python3-toml python3-wcwidth python3-webob 2026-02-25T22:20:35.193 INFO:teuthology.orchestra.run.trial188.stdout: python3-websocket python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2026-02-25T22:20:35.193 INFO:teuthology.orchestra.run.trial188.stdout: socat xmlstarlet 2026-02-25T22:20:35.389 INFO:teuthology.orchestra.run.trial188.stdout:0 upgraded, 87 newly installed, 0 to remove and 31 not upgraded. 2026-02-25T22:20:35.390 INFO:teuthology.orchestra.run.trial188.stdout:Need to get 702 MB of archives. 2026-02-25T22:20:35.390 INFO:teuthology.orchestra.run.trial188.stdout:After this operation, 2670 MB of additional disk space will be used. 2026-02-25T22:20:35.390 INFO:teuthology.orchestra.run.trial188.stdout:Get:1 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 20.3.0-5305-g0886691e-1jammy [3797 kB] 2026-02-25T22:20:35.552 INFO:teuthology.orchestra.run.trial188.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-25T22:20:35.848 INFO:teuthology.orchestra.run.trial188.stdout:Get:3 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 20.3.0-5305-g0886691e-1jammy [2910 kB] 2026-02-25T22:20:35.892 INFO:teuthology.orchestra.run.trial188.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-25T22:20:35.903 INFO:teuthology.orchestra.run.trial188.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-25T22:20:35.974 INFO:teuthology.orchestra.run.trial188.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-25T22:20:36.041 INFO:teuthology.orchestra.run.trial188.stdout:Get:7 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 20.3.0-5305-g0886691e-1jammy [882 kB] 2026-02-25T22:20:36.091 INFO:teuthology.orchestra.run.trial188.stdout:Get:8 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 20.3.0-5305-g0886691e-1jammy [359 kB] 2026-02-25T22:20:36.132 INFO:teuthology.orchestra.run.trial188.stdout:Get:9 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 20.3.0-5305-g0886691e-1jammy [32.8 kB] 2026-02-25T22:20:36.133 INFO:teuthology.orchestra.run.trial188.stdout:Get:10 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 20.3.0-5305-g0886691e-1jammy [225 kB] 2026-02-25T22:20:36.137 INFO:teuthology.orchestra.run.trial188.stdout:Get:11 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 20.3.0-5305-g0886691e-1jammy [88.4 kB] 2026-02-25T22:20:36.138 INFO:teuthology.orchestra.run.trial188.stdout:Get:12 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 20.3.0-5305-g0886691e-1jammy [338 kB] 2026-02-25T22:20:36.145 INFO:teuthology.orchestra.run.trial188.stdout:Get:13 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 20.3.0-5305-g0886691e-1jammy [9092 kB] 2026-02-25T22:20:36.175 INFO:teuthology.orchestra.run.trial188.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-25T22:20:36.186 INFO:teuthology.orchestra.run.trial188.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-25T22:20:36.213 INFO:teuthology.orchestra.run.trial188.stdout:Get:16 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-25T22:20:36.221 INFO:teuthology.orchestra.run.trial188.stdout:Get:17 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-25T22:20:36.223 INFO:teuthology.orchestra.run.trial188.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-25T22:20:36.224 INFO:teuthology.orchestra.run.trial188.stdout:Get:19 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-25T22:20:36.225 INFO:teuthology.orchestra.run.trial188.stdout:Get:20 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-25T22:20:36.241 INFO:teuthology.orchestra.run.trial188.stdout:Get:21 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-25T22:20:36.242 INFO:teuthology.orchestra.run.trial188.stdout:Get:22 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-25T22:20:36.248 INFO:teuthology.orchestra.run.trial188.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-25T22:20:36.315 INFO:teuthology.orchestra.run.trial188.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-25T22:20:36.315 INFO:teuthology.orchestra.run.trial188.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-25T22:20:36.315 INFO:teuthology.orchestra.run.trial188.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-25T22:20:36.316 INFO:teuthology.orchestra.run.trial188.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-25T22:20:36.316 INFO:teuthology.orchestra.run.trial188.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-25T22:20:36.316 INFO:teuthology.orchestra.run.trial188.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-25T22:20:36.316 INFO:teuthology.orchestra.run.trial188.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-25T22:20:36.320 INFO:teuthology.orchestra.run.trial188.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-25T22:20:36.320 INFO:teuthology.orchestra.run.trial188.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-25T22:20:36.390 INFO:teuthology.orchestra.run.trial188.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-25T22:20:36.392 INFO:teuthology.orchestra.run.trial188.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-25T22:20:36.392 INFO:teuthology.orchestra.run.trial188.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-25T22:20:36.393 INFO:teuthology.orchestra.run.trial188.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-25T22:20:36.418 INFO:teuthology.orchestra.run.trial188.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-25T22:20:36.420 INFO:teuthology.orchestra.run.trial188.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-25T22:20:36.420 INFO:teuthology.orchestra.run.trial188.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-25T22:20:36.478 INFO:teuthology.orchestra.run.trial188.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-25T22:20:36.478 INFO:teuthology.orchestra.run.trial188.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-25T22:20:36.479 INFO:teuthology.orchestra.run.trial188.stdout:Get:42 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-25T22:20:36.479 INFO:teuthology.orchestra.run.trial188.stdout:Get:43 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-25T22:20:36.479 INFO:teuthology.orchestra.run.trial188.stdout:Get:44 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-25T22:20:36.480 INFO:teuthology.orchestra.run.trial188.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-25T22:20:36.539 INFO:teuthology.orchestra.run.trial188.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-25T22:20:36.540 INFO:teuthology.orchestra.run.trial188.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-25T22:20:36.541 INFO:teuthology.orchestra.run.trial188.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-25T22:20:36.611 INFO:teuthology.orchestra.run.trial188.stdout:Get:49 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-25T22:20:36.614 INFO:teuthology.orchestra.run.trial188.stdout:Get:50 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-25T22:20:36.616 INFO:teuthology.orchestra.run.trial188.stdout:Get:51 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-25T22:20:36.620 INFO:teuthology.orchestra.run.trial188.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-25T22:20:36.620 INFO:teuthology.orchestra.run.trial188.stdout:Get:53 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 20.3.0-5305-g0886691e-1jammy [113 kB] 2026-02-25T22:20:36.621 INFO:teuthology.orchestra.run.trial188.stdout:Get:54 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 20.3.0-5305-g0886691e-1jammy [261 kB] 2026-02-25T22:20:36.622 INFO:teuthology.orchestra.run.trial188.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-25T22:20:36.623 INFO:teuthology.orchestra.run.trial188.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-25T22:20:36.623 INFO:teuthology.orchestra.run.trial188.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-25T22:20:36.626 INFO:teuthology.orchestra.run.trial188.stdout:Get:58 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 20.3.0-5305-g0886691e-1jammy [31.5 MB] 2026-02-25T22:20:36.683 INFO:teuthology.orchestra.run.trial188.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-25T22:20:36.684 INFO:teuthology.orchestra.run.trial188.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-25T22:20:36.690 INFO:teuthology.orchestra.run.trial188.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-25T22:20:36.756 INFO:teuthology.orchestra.run.trial188.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-25T22:20:36.757 INFO:teuthology.orchestra.run.trial188.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-25T22:20:36.758 INFO:teuthology.orchestra.run.trial188.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-25T22:20:36.759 INFO:teuthology.orchestra.run.trial188.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-25T22:20:37.846 INFO:teuthology.orchestra.run.trial188.stdout:Get:66 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 20.3.0-5305-g0886691e-1jammy [4983 kB] 2026-02-25T22:20:38.021 INFO:teuthology.orchestra.run.trial188.stdout:Get:67 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 20.3.0-5305-g0886691e-1jammy [293 kB] 2026-02-25T22:20:38.025 INFO:teuthology.orchestra.run.trial188.stdout:Get:68 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 20.3.0-5305-g0886691e-1jammy [127 kB] 2026-02-25T22:20:38.058 INFO:teuthology.orchestra.run.trial188.stdout:Get:69 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 20.3.0-5305-g0886691e-1jammy [922 kB] 2026-02-25T22:20:38.075 INFO:teuthology.orchestra.run.trial188.stdout:Get:70 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 20.3.0-5305-g0886691e-1jammy [7541 kB] 2026-02-25T22:20:38.364 INFO:teuthology.orchestra.run.trial188.stdout:Get:71 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd-classic amd64 20.3.0-5305-g0886691e-1jammy [468 MB] 2026-02-25T22:20:58.746 INFO:teuthology.orchestra.run.trial188.stdout:Get:72 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 20.3.0-5305-g0886691e-1jammy [180 kB] 2026-02-25T22:20:58.747 INFO:teuthology.orchestra.run.trial188.stdout:Get:73 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 20.3.0-5305-g0886691e-1jammy [14.0 kB] 2026-02-25T22:20:58.747 INFO:teuthology.orchestra.run.trial188.stdout:Get:74 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 20.3.0-5305-g0886691e-1jammy [1030 kB] 2026-02-25T22:20:58.752 INFO:teuthology.orchestra.run.trial188.stdout:Get:75 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 20.3.0-5305-g0886691e-1jammy [2437 kB] 2026-02-25T22:20:58.764 INFO:teuthology.orchestra.run.trial188.stdout:Get:76 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 20.3.0-5305-g0886691e-1jammy [1069 kB] 2026-02-25T22:20:58.799 INFO:teuthology.orchestra.run.trial188.stdout:Get:77 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 20.3.0-5305-g0886691e-1jammy [190 kB] 2026-02-25T22:20:58.808 INFO:teuthology.orchestra.run.trial188.stdout:Get:78 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 20.3.0-5305-g0886691e-1jammy [54.7 MB] 2026-02-25T22:21:00.297 INFO:teuthology.orchestra.run.trial188.stdout:Get:79 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 20.3.0-5305-g0886691e-1jammy [8626 kB] 2026-02-25T22:21:00.423 INFO:teuthology.orchestra.run.trial188.stdout:Get:80 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 20.3.0-5305-g0886691e-1jammy [14.2 kB] 2026-02-25T22:21:00.423 INFO:teuthology.orchestra.run.trial188.stdout:Get:81 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 20.3.0-5305-g0886691e-1jammy [68.6 MB] 2026-02-25T22:21:01.557 INFO:teuthology.orchestra.run.trial188.stdout:Get:82 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 20.3.0-5305-g0886691e-1jammy [137 kB] 2026-02-25T22:21:01.558 INFO:teuthology.orchestra.run.trial188.stdout:Get:83 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-daemon amd64 20.3.0-5305-g0886691e-1jammy [45.8 kB] 2026-02-25T22:21:01.559 INFO:teuthology.orchestra.run.trial188.stdout:Get:84 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-proxy2 amd64 20.3.0-5305-g0886691e-1jammy [32.4 kB] 2026-02-25T22:21:01.559 INFO:teuthology.orchestra.run.trial188.stdout:Get:85 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 20.3.0-5305-g0886691e-1jammy [44.2 kB] 2026-02-25T22:21:01.559 INFO:teuthology.orchestra.run.trial188.stdout:Get:86 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 20.3.0-5305-g0886691e-1jammy [18.3 MB] 2026-02-25T22:21:01.851 INFO:teuthology.orchestra.run.trial188.stdout:Get:87 https://3.chacra.ceph.com/r/ceph/wip-mon-colocate/0886691e8f7870d54dd61f71a1fe5f8750d0f7dd/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 20.3.0-5305-g0886691e-1jammy [94.1 kB] 2026-02-25T22:21:02.138 INFO:teuthology.orchestra.run.trial188.stdout:Fetched 702 MB in 27s (26.3 MB/s) 2026-02-25T22:21:02.158 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-25T22:21:02.324 INFO:teuthology.orchestra.run.trial188.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 202926 files and directories currently installed.) 2026-02-25T22:21:02.327 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../00-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-25T22:21:02.328 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-25T22:21:02.350 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-25T22:21:02.365 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../01-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-25T22:21:02.365 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-25T22:21:02.383 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-25T22:21:02.397 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../02-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-25T22:21:02.397 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-25T22:21:02.415 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-25T22:21:02.429 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../03-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-25T22:21:02.433 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T22:21:02.466 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-25T22:21:02.476 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../04-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-25T22:21:02.476 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T22:21:02.492 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-25T22:21:02.502 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../05-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-25T22:21:02.503 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T22:21:02.523 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-25T22:21:02.533 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../06-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-25T22:21:02.534 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-25T22:21:02.550 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package librados2. 2026-02-25T22:21:02.560 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../07-librados2_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:02.560 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking librados2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:02.603 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libnbd0. 2026-02-25T22:21:02.613 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../08-libnbd0_1.10.5-1_amd64.deb ... 2026-02-25T22:21:02.613 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-25T22:21:02.629 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package librbd1. 2026-02-25T22:21:02.638 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../09-librbd1_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:02.639 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking librbd1 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:02.677 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libcephfs2. 2026-02-25T22:21:02.687 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../10-libcephfs2_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:02.688 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libcephfs2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:02.708 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-rados. 2026-02-25T22:21:02.718 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../11-python3-rados_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:02.718 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-rados (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:02.736 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-25T22:21:02.746 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../12-python3-ceph-argparse_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:21:02.746 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-ceph-argparse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:02.761 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-cephfs. 2026-02-25T22:21:02.771 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../13-python3-cephfs_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:02.771 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-cephfs (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:02.788 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-25T22:21:02.798 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../14-python3-ceph-common_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:21:02.799 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-ceph-common (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:02.817 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-25T22:21:02.827 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../15-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-25T22:21:02.827 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-25T22:21:02.844 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-prettytable. 2026-02-25T22:21:02.854 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../16-python3-prettytable_2.5.0-2_all.deb ... 2026-02-25T22:21:02.855 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-25T22:21:02.869 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-rbd. 2026-02-25T22:21:02.879 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../17-python3-rbd_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:02.880 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-rbd (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:02.897 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-25T22:21:02.907 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../18-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-25T22:21:02.908 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-25T22:21:02.926 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package librgw2. 2026-02-25T22:21:02.936 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../19-librgw2_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:02.937 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking librgw2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:03.036 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-rgw. 2026-02-25T22:21:03.046 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../20-python3-rgw_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:03.047 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-rgw (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:03.062 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-25T22:21:03.072 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../21-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-25T22:21:03.072 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-25T22:21:03.087 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libradosstriper1. 2026-02-25T22:21:03.098 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../22-libradosstriper1_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:03.098 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libradosstriper1 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:03.115 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-common. 2026-02-25T22:21:03.125 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../23-ceph-common_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:03.126 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-common (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:03.387 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-base. 2026-02-25T22:21:03.397 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../24-ceph-base_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:03.402 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-base (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:03.462 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-25T22:21:03.472 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../25-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-25T22:21:03.473 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-25T22:21:03.488 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-cheroot. 2026-02-25T22:21:03.498 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../26-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-25T22:21:03.498 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-25T22:21:03.516 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-25T22:21:03.526 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../27-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-25T22:21:03.526 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-25T22:21:03.541 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-25T22:21:03.551 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../28-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-25T22:21:03.552 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-25T22:21:03.566 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-25T22:21:03.576 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../29-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-25T22:21:03.577 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-25T22:21:03.591 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-tempora. 2026-02-25T22:21:03.607 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../30-python3-tempora_4.1.2-1_all.deb ... 2026-02-25T22:21:03.607 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-25T22:21:03.617 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-portend. 2026-02-25T22:21:03.627 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../31-python3-portend_3.0.0-1_all.deb ... 2026-02-25T22:21:03.627 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-25T22:21:03.643 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-25T22:21:03.653 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../32-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-25T22:21:03.653 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-25T22:21:03.668 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-25T22:21:03.678 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../33-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-25T22:21:03.679 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-25T22:21:03.702 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-natsort. 2026-02-25T22:21:03.712 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../34-python3-natsort_8.0.2-1_all.deb ... 2026-02-25T22:21:03.713 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-25T22:21:03.729 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-25T22:21:03.739 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../35-ceph-mgr-modules-core_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:21:03.739 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-mgr-modules-core (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:03.770 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-25T22:21:03.780 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../36-libsqlite3-mod-ceph_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:03.781 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libsqlite3-mod-ceph (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:03.796 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-mgr. 2026-02-25T22:21:03.806 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../37-ceph-mgr_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:03.806 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-mgr (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:03.827 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-mon. 2026-02-25T22:21:03.837 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../38-ceph-mon_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:03.837 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-mon (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:03.903 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-25T22:21:03.913 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../39-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-25T22:21:03.913 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-25T22:21:03.930 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-osd-classic. 2026-02-25T22:21:03.939 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../40-ceph-osd-classic_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:03.940 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-osd-classic (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:07.832 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-osd. 2026-02-25T22:21:07.848 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../41-ceph-osd_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:07.848 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-osd (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:07.869 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph. 2026-02-25T22:21:07.885 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../42-ceph_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:07.885 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:07.902 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-fuse. 2026-02-25T22:21:07.917 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../43-ceph-fuse_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:07.917 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-fuse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:07.940 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-mds. 2026-02-25T22:21:07.951 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../44-ceph-mds_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:07.952 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-mds (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:07.985 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package cephadm. 2026-02-25T22:21:07.995 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../45-cephadm_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:07.996 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking cephadm (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:08.013 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-25T22:21:08.023 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../46-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-25T22:21:08.023 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-25T22:21:08.046 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-25T22:21:08.056 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../47-ceph-mgr-cephadm_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:21:08.056 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-mgr-cephadm (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:08.078 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-25T22:21:08.089 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../48-python3-repoze.lru_0.7-2_all.deb ... 2026-02-25T22:21:08.089 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-25T22:21:08.105 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-routes. 2026-02-25T22:21:08.115 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../49-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-25T22:21:08.116 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-25T22:21:08.135 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-25T22:21:08.146 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../50-ceph-mgr-dashboard_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:21:08.146 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-mgr-dashboard (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:08.847 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-25T22:21:08.859 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../51-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-25T22:21:08.859 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-25T22:21:08.899 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-joblib. 2026-02-25T22:21:08.910 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../52-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-25T22:21:08.910 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-25T22:21:08.937 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-25T22:21:08.948 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../53-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-25T22:21:08.948 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-25T22:21:08.964 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-sklearn. 2026-02-25T22:21:08.975 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../54-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-25T22:21:08.975 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-25T22:21:09.060 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-25T22:21:09.071 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../55-ceph-mgr-diskprediction-local_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:21:09.072 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-mgr-diskprediction-local (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:09.228 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-cachetools. 2026-02-25T22:21:09.239 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../56-python3-cachetools_5.0.0-1_all.deb ... 2026-02-25T22:21:09.239 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-25T22:21:09.255 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-rsa. 2026-02-25T22:21:09.266 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../57-python3-rsa_4.8-1_all.deb ... 2026-02-25T22:21:09.267 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-25T22:21:09.285 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-google-auth. 2026-02-25T22:21:09.295 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../58-python3-google-auth_1.5.1-3_all.deb ... 2026-02-25T22:21:09.296 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-25T22:21:09.315 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-25T22:21:09.325 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../59-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-25T22:21:09.325 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-25T22:21:09.341 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-websocket. 2026-02-25T22:21:09.352 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../60-python3-websocket_1.2.3-1_all.deb ... 2026-02-25T22:21:09.353 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-25T22:21:09.371 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-25T22:21:09.381 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../61-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-25T22:21:09.381 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-25T22:21:09.480 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-25T22:21:09.491 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../62-ceph-mgr-k8sevents_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:21:09.492 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-mgr-k8sevents (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:09.506 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-25T22:21:09.517 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../63-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-25T22:21:09.517 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-25T22:21:09.534 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-25T22:21:09.545 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../64-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-25T22:21:09.545 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-25T22:21:09.561 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package jq. 2026-02-25T22:21:09.571 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../65-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-25T22:21:09.572 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-25T22:21:09.586 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package socat. 2026-02-25T22:21:09.597 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../66-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-25T22:21:09.598 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-25T22:21:09.618 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package xmlstarlet. 2026-02-25T22:21:09.629 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../67-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-25T22:21:09.629 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-25T22:21:09.665 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-test. 2026-02-25T22:21:09.676 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../68-ceph-test_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:09.676 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-test (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:10.323 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package ceph-volume. 2026-02-25T22:21:10.333 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../69-ceph-volume_20.3.0-5305-g0886691e-1jammy_all.deb ... 2026-02-25T22:21:10.334 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking ceph-volume (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:10.357 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libcephfs-daemon. 2026-02-25T22:21:10.368 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../70-libcephfs-daemon_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:10.369 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libcephfs-daemon (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:10.384 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libcephfs-proxy2. 2026-02-25T22:21:10.394 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../71-libcephfs-proxy2_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:10.395 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libcephfs-proxy2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:10.410 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-25T22:21:10.421 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../72-libcephfs-dev_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:10.421 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking libcephfs-dev (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:10.437 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package nvme-cli. 2026-02-25T22:21:10.448 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../73-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-25T22:21:10.448 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-25T22:21:10.479 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-25T22:21:10.490 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../74-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-25T22:21:10.490 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-25T22:21:10.520 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-25T22:21:10.530 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../75-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-25T22:21:10.531 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-25T22:21:10.547 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-pluggy. 2026-02-25T22:21:10.557 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../76-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-25T22:21:10.557 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-25T22:21:10.573 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-psutil. 2026-02-25T22:21:10.584 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../77-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-25T22:21:10.584 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-25T22:21:10.603 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-py. 2026-02-25T22:21:10.613 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../78-python3-py_1.10.0-1_all.deb ... 2026-02-25T22:21:10.614 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-25T22:21:10.635 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-pygments. 2026-02-25T22:21:10.645 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../79-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-25T22:21:10.646 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-25T22:21:10.687 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-toml. 2026-02-25T22:21:10.697 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../80-python3-toml_0.10.2-1_all.deb ... 2026-02-25T22:21:10.698 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-25T22:21:10.713 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-pytest. 2026-02-25T22:21:10.724 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../81-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-25T22:21:10.724 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-25T22:21:10.746 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-simplejson. 2026-02-25T22:21:10.757 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../82-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-25T22:21:10.757 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-25T22:21:10.783 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package python3-webob. 2026-02-25T22:21:10.793 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../83-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-25T22:21:10.794 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-25T22:21:10.811 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-25T22:21:10.821 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../84-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-25T22:21:10.821 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-25T22:21:10.891 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package radosgw. 2026-02-25T22:21:10.902 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../85-radosgw_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:10.903 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking radosgw (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:11.064 INFO:teuthology.orchestra.run.trial188.stdout:Selecting previously unselected package rbd-fuse. 2026-02-25T22:21:11.074 INFO:teuthology.orchestra.run.trial188.stdout:Preparing to unpack .../86-rbd-fuse_20.3.0-5305-g0886691e-1jammy_amd64.deb ... 2026-02-25T22:21:11.075 INFO:teuthology.orchestra.run.trial188.stdout:Unpacking rbd-fuse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:11.104 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-25T22:21:11.172 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-25T22:21:11.173 INFO:teuthology.orchestra.run.trial188.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-25T22:21:11.247 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-25T22:21:11.518 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-25T22:21:11.907 INFO:teuthology.orchestra.run.trial188.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-25T22:21:11.918 INFO:teuthology.orchestra.run.trial188.stdout:Setting up cephadm (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:11.959 INFO:teuthology.orchestra.run.trial188.stdout:Adding system user cephadm....done 2026-02-25T22:21:11.964 INFO:teuthology.orchestra.run.trial188.stdout:usermod: no changes 2026-02-25T22:21:11.969 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-25T22:21:12.033 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-25T22:21:12.034 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-25T22:21:12.105 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-25T22:21:12.176 INFO:teuthology.orchestra.run.trial188.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-25T22:21:12.177 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-25T22:21:12.274 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-25T22:21:12.404 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-25T22:21:12.478 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-25T22:21:12.554 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-ceph-argparse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:12.623 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-25T22:21:12.623 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-25T22:21:12.625 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-25T22:21:12.626 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-25T22:21:12.626 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-25T22:21:12.748 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-25T22:21:12.826 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libcephfs-proxy2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:12.827 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-25T22:21:12.907 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-25T22:21:12.997 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-25T22:21:13.254 INFO:teuthology.orchestra.run.trial188.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-25T22:21:13.255 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-25T22:21:13.354 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-25T22:21:13.486 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-25T22:21:13.569 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-25T22:21:13.640 INFO:teuthology.orchestra.run.trial188.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-25T22:21:13.641 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-ceph-common (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:13.732 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-25T22:21:14.248 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T22:21:14.252 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-25T22:21:14.332 INFO:teuthology.orchestra.run.trial188.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-25T22:21:14.333 INFO:teuthology.orchestra.run.trial188.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-25T22:21:14.334 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-25T22:21:14.410 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-25T22:21:14.483 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T22:21:14.484 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-25T22:21:14.562 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-25T22:21:14.638 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-25T22:21:14.718 INFO:teuthology.orchestra.run.trial188.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-25T22:21:14.719 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-25T22:21:14.802 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-25T22:21:14.803 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-25T22:21:14.880 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-25T22:21:14.965 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-25T22:21:15.038 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-25T22:21:15.039 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-25T22:21:15.175 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-25T22:21:15.248 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-25T22:21:15.250 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-25T22:21:15.334 INFO:teuthology.orchestra.run.trial188.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-25T22:21:15.335 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-25T22:21:15.468 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-25T22:21:15.469 INFO:teuthology.orchestra.run.trial188.stdout:Setting up librados2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:15.470 INFO:teuthology.orchestra.run.trial188.stdout:Setting up librgw2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:15.471 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libsqlite3-mod-ceph (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:15.472 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-25T22:21:16.010 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libcephfs2 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:16.011 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libradosstriper1 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:16.012 INFO:teuthology.orchestra.run.trial188.stdout:Setting up librbd1 (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:16.013 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-mgr-modules-core (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:16.014 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-fuse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:16.086 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-25T22:21:16.087 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-25T22:21:16.461 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libcephfs-dev (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:16.462 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-rados (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:16.463 INFO:teuthology.orchestra.run.trial188.stdout:Setting up libcephfs-daemon (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:16.464 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-rbd (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:16.465 INFO:teuthology.orchestra.run.trial188.stdout:Setting up rbd-fuse (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:16.466 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-rgw (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:16.467 INFO:teuthology.orchestra.run.trial188.stdout:Setting up python3-cephfs (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:16.468 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-common (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:16.496 INFO:teuthology.orchestra.run.trial188.stdout:Adding group ceph....done 2026-02-25T22:21:16.532 INFO:teuthology.orchestra.run.trial188.stdout:Adding system user ceph....done 2026-02-25T22:21:16.542 INFO:teuthology.orchestra.run.trial188.stdout:Setting system user ceph properties....done 2026-02-25T22:21:16.549 INFO:teuthology.orchestra.run.trial188.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-25T22:21:16.636 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-25T22:21:16.911 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-25T22:21:17.317 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-test (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:17.318 INFO:teuthology.orchestra.run.trial188.stdout:Setting up radosgw (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:17.565 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-25T22:21:17.565 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-25T22:21:17.914 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-base (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:18.028 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-25T22:21:18.438 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-mds (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:18.496 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-25T22:21:18.497 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-25T22:21:18.842 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-mgr (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:18.920 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-25T22:21:18.920 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-25T22:21:19.305 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-mgr-k8sevents (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:19.306 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-osd-classic (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:19.321 INFO:teuthology.orchestra.run.trial188.stdout:update-alternatives: using /usr/bin/ceph-osd-classic to provide /usr/bin/ceph-osd (ceph-osd) in auto mode 2026-02-25T22:21:19.331 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-mgr-diskprediction-local (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:19.343 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-mon (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:19.417 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-25T22:21:19.417 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-25T22:21:19.808 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-mgr-cephadm (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:19.818 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-mgr-dashboard (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:19.829 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-osd (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:19.905 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-25T22:21:19.906 INFO:teuthology.orchestra.run.trial188.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-25T22:21:20.310 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:20.311 INFO:teuthology.orchestra.run.trial188.stdout:Setting up ceph-volume (20.3.0-5305-g0886691e-1jammy) ... 2026-02-25T22:21:20.422 INFO:teuthology.orchestra.run.trial188.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-25T22:21:20.442 INFO:teuthology.orchestra.run.trial188.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.13) ... 2026-02-25T22:21:21.077 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:21:21.077 INFO:teuthology.orchestra.run.trial188.stdout:Running kernel seems to be up-to-date. 2026-02-25T22:21:21.078 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:21:21.078 INFO:teuthology.orchestra.run.trial188.stdout:The processor microcode seems to be up-to-date. 2026-02-25T22:21:21.078 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:21:21.078 INFO:teuthology.orchestra.run.trial188.stdout:No services need to be restarted. 2026-02-25T22:21:21.078 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:21:21.078 INFO:teuthology.orchestra.run.trial188.stdout:No containers need to be restarted. 2026-02-25T22:21:21.078 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:21:21.078 INFO:teuthology.orchestra.run.trial188.stdout:No user sessions are running outdated binaries. 2026-02-25T22:21:21.078 INFO:teuthology.orchestra.run.trial188.stdout: 2026-02-25T22:21:21.078 INFO:teuthology.orchestra.run.trial188.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-25T22:21:21.919 INFO:teuthology.orchestra.run.trial188.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-25T22:21:21.928 DEBUG:teuthology.parallel:result is None 2026-02-25T22:21:21.929 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=0886691e8f7870d54dd61f71a1fe5f8750d0f7dd 2026-02-25T22:21:22.096 DEBUG:teuthology.orchestra.run.trial188:> dpkg-query -W -f '${Version}' ceph 2026-02-25T22:21:22.108 INFO:teuthology.orchestra.run.trial188.stdout:20.3.0-5305-g0886691e-1jammy 2026-02-25T22:21:22.109 INFO:teuthology.packaging:The installed version of ceph is 20.3.0-5305-g0886691e-1jammy 2026-02-25T22:21:22.109 INFO:teuthology.task.install:The correct ceph version 20.3.0-5305-g0886691e-1jammy is installed. 2026-02-25T22:21:22.112 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-25T22:21:22.113 DEBUG:teuthology.orchestra.run.trial188:> set -ex 2026-02-25T22:21:22.113 DEBUG:teuthology.orchestra.run.trial188:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-25T22:21:22.160 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-25T22:21:22.161 DEBUG:teuthology.orchestra.run.trial188:> set -ex 2026-02-25T22:21:22.161 DEBUG:teuthology.orchestra.run.trial188:> sudo dd of=/usr/bin/daemon-helper 2026-02-25T22:21:22.211 DEBUG:teuthology.orchestra.run.trial188:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-25T22:21:22.263 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-25T22:21:22.264 DEBUG:teuthology.orchestra.run.trial188:> set -ex 2026-02-25T22:21:22.264 DEBUG:teuthology.orchestra.run.trial188:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-25T22:21:22.313 DEBUG:teuthology.orchestra.run.trial188:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-25T22:21:22.367 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-25T22:21:22.368 DEBUG:teuthology.orchestra.run.trial188:> set -ex 2026-02-25T22:21:22.368 DEBUG:teuthology.orchestra.run.trial188:> sudo dd of=/usr/bin/stdin-killer 2026-02-25T22:21:22.420 DEBUG:teuthology.orchestra.run.trial188:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-25T22:21:22.472 INFO:teuthology.run_tasks:Running task workunit... 2026-02-25T22:21:22.504 INFO:tasks.workunit:Pulling workunits from ref 61d4431d3ec879bd98659388b3655acf3d392418 2026-02-25T22:21:22.504 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-25T22:21:22.504 INFO:tasks.workunit:timeout=3h 2026-02-25T22:21:22.505 INFO:tasks.workunit:cleanup=True 2026-02-25T22:21:22.505 DEBUG:teuthology.orchestra.run.trial188:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-25T22:21:22.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-25T22:21:22.519 INFO:teuthology.orchestra.run.trial188.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-25T22:21:22.520 DEBUG:teuthology.orchestra.run.trial188:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-25T22:21:22.567 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-25T22:21:22.567 DEBUG:teuthology.orchestra.run.trial188:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-25T22:21:22.611 DEBUG:teuthology.orchestra.run.trial188:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/Ericmzhang/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 61d4431d3ec879bd98659388b3655acf3d392418 2026-02-25T22:21:22.662 INFO:tasks.workunit.client.0.trial188.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-25T22:21:40.211 INFO:tasks.workunit.client.0.trial188.stderr:Note: switching to '61d4431d3ec879bd98659388b3655acf3d392418'. 2026-02-25T22:21:40.211 INFO:tasks.workunit.client.0.trial188.stderr: 2026-02-25T22:21:40.212 INFO:tasks.workunit.client.0.trial188.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-25T22:21:40.212 INFO:tasks.workunit.client.0.trial188.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-25T22:21:40.212 INFO:tasks.workunit.client.0.trial188.stderr:state without impacting any branches by switching back to a branch. 2026-02-25T22:21:40.212 INFO:tasks.workunit.client.0.trial188.stderr: 2026-02-25T22:21:40.212 INFO:tasks.workunit.client.0.trial188.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-25T22:21:40.212 INFO:tasks.workunit.client.0.trial188.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-25T22:21:40.212 INFO:tasks.workunit.client.0.trial188.stderr: 2026-02-25T22:21:40.212 INFO:tasks.workunit.client.0.trial188.stderr: git switch -c 2026-02-25T22:21:40.212 INFO:tasks.workunit.client.0.trial188.stderr: 2026-02-25T22:21:40.212 INFO:tasks.workunit.client.0.trial188.stderr:Or undo this operation with: 2026-02-25T22:21:40.212 INFO:tasks.workunit.client.0.trial188.stderr: 2026-02-25T22:21:40.213 INFO:tasks.workunit.client.0.trial188.stderr: git switch - 2026-02-25T22:21:40.213 INFO:tasks.workunit.client.0.trial188.stderr: 2026-02-25T22:21:40.213 INFO:tasks.workunit.client.0.trial188.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-25T22:21:40.213 INFO:tasks.workunit.client.0.trial188.stderr: 2026-02-25T22:21:40.213 INFO:tasks.workunit.client.0.trial188.stderr:HEAD is now at 61d4431d3ec changed test script name so it runs first 2026-02-25T22:21:40.214 DEBUG:teuthology.orchestra.run.trial188:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-25T22:21:40.261 DEBUG:teuthology.orchestra.run.trial188:> set -ex 2026-02-25T22:21:40.261 DEBUG:teuthology.orchestra.run.trial188:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-25T22:21:40.308 INFO:tasks.workunit:Running workunits matching misc on client.0... 2026-02-25T22:21:40.309 INFO:tasks.workunit:Running workunit misc/mclock-config.sh... 2026-02-25T22:21:40.310 DEBUG:teuthology.orchestra.run.trial188:workunit test misc/mclock-config.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61d4431d3ec879bd98659388b3655acf3d392418 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh 2026-02-25T22:21:40.361 INFO:tasks.workunit.client.0.trial188.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T22:21:40.366 INFO:tasks.workunit.client.0.trial188.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T22:21:40.366 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:21:40.366 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:21:40.366 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T22:21:40.366 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T22:21:40.366 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T22:21:40.366 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T22:21:40.366 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T22:21:40.366 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T22:21:40.366 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/mclock-config 2026-02-25T22:21:40.367 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:21: run: local dir=td/mclock-config 2026-02-25T22:21:40.367 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:22: run: shift 2026-02-25T22:21:40.367 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: export CEPH_MON=127.0.0.1:7124 2026-02-25T22:21:40.367 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:24: run: CEPH_MON=127.0.0.1:7124 2026-02-25T22:21:40.367 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:25: run: export CEPH_ARGS 2026-02-25T22:21:40.367 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: uuidgen 2026-02-25T22:21:40.368 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:26: run: CEPH_ARGS+='--fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none ' 2026-02-25T22:21:40.368 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:27: run: CEPH_ARGS+='--mon-host=127.0.0.1:7124 ' 2026-02-25T22:21:40.368 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:28: run: CEPH_ARGS+='--debug-mclock 20 ' 2026-02-25T22:21:40.368 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: set 2026-02-25T22:21:40.368 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-25T22:21:40.370 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:30: run: local 'funcs=TEST_backfill_limit_adjustment_mclock 2026-02-25T22:21:40.370 INFO:tasks.workunit.client.0.trial188.stderr:TEST_profile_builtin_to_custom 2026-02-25T22:21:40.370 INFO:tasks.workunit.client.0.trial188.stderr:TEST_profile_custom_to_builtin 2026-02-25T22:21:40.370 INFO:tasks.workunit.client.0.trial188.stderr:TEST_recovery_limit_adjustment_mclock' 2026-02-25T22:21:40.370 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-25T22:21:40.370 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-25T22:21:40.370 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T22:21:40.370 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T22:21:40.370 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:21:40.370 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:40.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:21:40.371 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:40.371 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:40.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:40.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:40.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:40.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:40.372 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:40.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:40.373 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:40.374 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:40.374 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:40.375 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:40.375 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:40.375 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:40.376 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:40.376 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:40.376 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:40.378 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:40.378 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:21:40.379 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:40.380 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:40.380 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:40.380 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:21:40.381 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:40.381 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:40.381 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T22:21:40.382 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:21:40.382 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:40.382 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:40.382 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11509 2026-02-25T22:21:40.384 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:21:40.386 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T22:21:40.386 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T22:21:40.386 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:21:40.386 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T22:21:40.386 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_backfill_limit_adjustment_mclock td/mclock-config 2026-02-25T22:21:40.386 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:236: TEST_backfill_limit_adjustment_mclock: local dir=td/mclock-config 2026-02-25T22:21:40.386 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:238: TEST_backfill_limit_adjustment_mclock: setup td/mclock-config 2026-02-25T22:21:40.386 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T22:21:40.386 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T22:21:40.386 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:21:40.386 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:40.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:21:40.387 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:40.387 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:40.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:40.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:40.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:40.388 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:40.388 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:40.389 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:40.389 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:40.390 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:40.390 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:40.391 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:40.391 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:40.392 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:40.392 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:40.392 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:40.393 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:40.394 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:40.394 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:21:40.396 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:40.396 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:40.396 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:40.396 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:21:40.397 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:40.397 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:40.398 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T22:21:40.399 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:21:40.399 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:40.399 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:40.399 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11509 2026-02-25T22:21:40.401 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:21:40.402 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:21:40.402 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:21:40.402 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T22:21:40.402 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:239: TEST_backfill_limit_adjustment_mclock: run_mon td/mclock-config a 2026-02-25T22:21:40.402 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-25T22:21:40.402 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:21:40.402 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:21:40.403 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:21:40.403 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-25T22:21:40.403 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-25T22:21:40.441 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:21:40.441 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:40.441 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:40.441 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:40.442 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:40.442 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:40.442 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:40.442 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:21:40.473 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:21:40.474 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:21:40.474 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:40.474 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:40.474 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:21:40.475 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:21:40.476 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:40.476 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:40.476 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:40.476 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:40.476 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:40.476 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:40.476 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-mon.a.asok 2026-02-25T22:21:40.476 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:40.477 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11509/ceph-mon.a.asok config get fsid 2026-02-25T22:21:40.532 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:21:40.532 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:40.532 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:40.532 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:21:40.532 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:21:40.533 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:40.533 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:40.533 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:40.533 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:40.534 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:40.534 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:40.534 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-mon.a.asok 2026-02-25T22:21:40.534 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:40.534 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11509/ceph-mon.a.asok config get mon_host 2026-02-25T22:21:40.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:240: TEST_backfill_limit_adjustment_mclock: run_mgr td/mclock-config x 2026-02-25T22:21:40.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-25T22:21:40.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:21:40.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:21:40.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:21:40.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-25T22:21:40.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:21:40.701 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:21:40.701 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:40.701 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:40.702 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:40.702 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:40.702 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:40.702 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:40.702 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:40.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:40.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:242: TEST_backfill_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-25T22:21:40.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-25T22:21:40.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:21:40.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:21:40.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:21:40.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-25T22:21:40.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-25T22:21:40.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:21:40.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:21:40.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:21:40.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-25T22:21:40.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-25T22:21:40.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:21:40.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:21:40.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-25T22:21:40.725 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:21:40.726 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:40.726 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:40.726 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:40.726 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:40.726 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:40.726 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:40.726 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:40.726 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:21:40.726 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:21:40.726 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:21:40.726 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-25T22:21:40.726 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-25T22:21:40.726 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:21:40.727 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:21:40.727 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:21:40.727 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:21:40.727 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:21:40.727 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-25T22:21:40.727 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-25T22:21:40.728 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:21:40.728 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ad6da4e6-32e6-476c-b5ca-ca14c7f3f5e8 2026-02-25T22:21:40.728 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 ad6da4e6-32e6-476c-b5ca-ca14c7f3f5e8' 2026-02-25T22:21:40.728 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 ad6da4e6-32e6-476c-b5ca-ca14c7f3f5e8 2026-02-25T22:21:40.728 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:21:40.737 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD0dZ9pkxe9KxAAIug7+W8r4ERizffnA40qLQ== 2026-02-25T22:21:40.737 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD0dZ9pkxe9KxAAIug7+W8r4ERizffnA40qLQ=="}' 2026-02-25T22:21:40.737 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ad6da4e6-32e6-476c-b5ca-ca14c7f3f5e8 -i td/mclock-config/0/new.json 2026-02-25T22:21:40.847 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:21:40.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-25T22:21:40.855 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQD0dZ9pkxe9KxAAIug7+W8r4ERizffnA40qLQ== --osd-uuid ad6da4e6-32e6-476c-b5ca-ca14c7f3f5e8 2026-02-25T22:21:40.871 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:40.862+0000 7f789b433a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:40.873 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:40.866+0000 7f789b433a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:40.874 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:40.866+0000 7f789b433a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:40.874 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:40.866+0000 7f789b433a40 -1 bdev(0x55c576afa400 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:21:40.874 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:40.866+0000 7f789b433a40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-25T22:21:41.607 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-25T22:21:41.608 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:21:41.608 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:21:41.609 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:21:41.609 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:21:41.728 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:21:41.729 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:21:41.729 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-25T22:21:41.729 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:21:41.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:21:41.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:21:41.750 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:41.742+0000 7f9289e8ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:41.752 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:41.746+0000 7f9289e8ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:41.753 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:41.746+0000 7f9289e8ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:41.837 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:21:41.837 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:21:41.837 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:21:41.837 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:21:41.837 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:21:41.837 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:41.838 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:21:41.838 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:21:41.838 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:41.838 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:41.948 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:42.321 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:42.314+0000 7f9289e8ea40 -1 Falling back to public interface 2026-02-25T22:21:42.950 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:42.950 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:42.950 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:21:42.951 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:21:42.951 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:42.951 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:43.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:43.377 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:43.370+0000 7f9289e8ea40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:21:44.061 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:44.061 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:44.061 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:21:44.062 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:21:44.062 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:44.062 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:44.277 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3095544368,v1:127.0.0.1:6801/3095544368] [v2:127.0.0.1:6802/3095544368,v1:127.0.0.1:6803/3095544368] exists,up ad6da4e6-32e6-476c-b5ca-ca14c7f3f5e8 2026-02-25T22:21:44.277 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:21:44.278 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:21:44.278 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:21:44.278 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-25T22:21:44.278 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T22:21:44.279 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T22:21:44.279 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:44.280 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:44.280 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:44.280 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:44.280 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:44.280 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:44.280 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T22:21:44.280 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T22:21:44.340 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:244: TEST_backfill_limit_adjustment_mclock: local backfills=1 2026-02-25T22:21:44.340 INFO:tasks.workunit.client.0.trial188.stdout:osd_max_backfills: 1 2026-02-25T22:21:44.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:246: TEST_backfill_limit_adjustment_mclock: echo 'osd_max_backfills: 1' 2026-02-25T22:21:44.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:251: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2026-02-25T22:21:44.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:252: TEST_backfill_limit_adjustment_mclock: sleep 2 2026-02-25T22:21:46.564 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-25T22:21:46.564 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T22:21:46.565 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T22:21:46.565 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:46.565 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:46.566 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:46.566 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:46.566 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:46.566 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:46.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T22:21:46.567 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T22:21:46.626 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:254: TEST_backfill_limit_adjustment_mclock: local max_backfills=1 2026-02-25T22:21:46.626 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:255: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-25T22:21:46.627 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-25T22:21:46.627 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T22:21:46.628 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T22:21:46.628 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:46.628 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:46.628 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:46.628 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:46.628 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:46.628 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:46.629 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T22:21:46.629 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok dump_recovery_reservations 2026-02-25T22:21:46.686 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:259: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2026-02-25T22:21:46.686 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:260: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-25T22:21:46.687 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-25T22:21:46.687 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T22:21:46.688 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T22:21:46.688 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:46.688 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:46.688 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:46.688 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:46.688 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:46.689 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:46.689 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T22:21:46.689 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok dump_recovery_reservations 2026-02-25T22:21:46.746 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:262: TEST_backfill_limit_adjustment_mclock: max_backfills=1 2026-02-25T22:21:46.746 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:263: TEST_backfill_limit_adjustment_mclock: test 1 = 1 2026-02-25T22:21:46.746 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:267: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2026-02-25T22:21:46.969 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:268: TEST_backfill_limit_adjustment_mclock: ceph config set osd.0 osd_max_backfills 20 2026-02-25T22:21:47.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:269: TEST_backfill_limit_adjustment_mclock: sleep 2 2026-02-25T22:21:49.209 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-25T22:21:49.209 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T22:21:49.210 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T22:21:49.210 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:49.210 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:49.211 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:49.211 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:49.211 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:49.211 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:49.211 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T22:21:49.211 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T22:21:49.273 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:271: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-25T22:21:49.273 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:272: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-25T22:21:49.274 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-25T22:21:49.274 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T22:21:49.275 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T22:21:49.275 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:49.275 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:49.275 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:49.275 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:49.275 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:49.275 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:49.275 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T22:21:49.276 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok dump_recovery_reservations 2026-02-25T22:21:49.333 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:276: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-25T22:21:49.333 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:277: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-25T22:21:49.334 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-25T22:21:49.334 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T22:21:49.335 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T22:21:49.335 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:49.335 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:49.335 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:49.335 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:49.335 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:49.335 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:49.336 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T22:21:49.336 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok dump_recovery_reservations 2026-02-25T22:21:49.395 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:279: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-25T22:21:49.395 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:280: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-25T22:21:49.395 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:284: TEST_backfill_limit_adjustment_mclock: kill_daemons td/mclock-config TERM osd 2026-02-25T22:21:49.395 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:49.395 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:49.395 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:49.396 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:49.396 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:49.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:49.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:285: TEST_backfill_limit_adjustment_mclock: ceph osd down 0 2026-02-25T22:21:49.914 INFO:tasks.workunit.client.0.trial188.stderr:osd.0 is already down. 2026-02-25T22:21:49.924 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:286: TEST_backfill_limit_adjustment_mclock: wait_for_osd down 0 2026-02-25T22:21:49.924 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2026-02-25T22:21:49.924 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:21:49.924 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:21:49.925 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:21:49.925 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:49.925 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:21:49.925 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:21:49.926 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:49.926 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2026-02-25T22:21:50.135 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/3095544368,v1:127.0.0.1:6801/3095544368] [v2:127.0.0.1:6802/3095544368,v1:127.0.0.1:6803/3095544368] exists ad6da4e6-32e6-476c-b5ca-ca14c7f3f5e8 2026-02-25T22:21:50.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:21:50.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:21:50.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:21:50.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:287: TEST_backfill_limit_adjustment_mclock: activate_osd td/mclock-config 0 --osd-op-queue=mclock_scheduler 2026-02-25T22:21:50.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/mclock-config 2026-02-25T22:21:50.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:21:50.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-25T22:21:50.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:21:50.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/mclock-config/0 2026-02-25T22:21:50.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-25T22:21:50.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:21:50.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:21:50.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:21:50.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-25T22:21:50.140 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-25T22:21:50.140 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:21:50.140 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:21:50.140 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-25T22:21:50.140 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:21:50.140 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:50.140 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:50.140 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:50.140 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:50.140 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:50.140 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:50.140 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:50.140 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:21:50.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-25T22:21:50.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-25T22:21:50.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:21:50.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:21:50.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:21:50.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:21:50.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:21:50.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+=--osd-op-queue=mclock_scheduler 2026-02-25T22:21:50.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/mclock-config/0 2026-02-25T22:21:50.141 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:21:50.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-25T22:21:50.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --debug-osd=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler 2026-02-25T22:21:50.143 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/mclock-config/0/whoami 2026-02-25T22:21:50.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-25T22:21:50.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:21:50.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:21:50.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:21:50.157 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:50.150+0000 7f64a8180a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:50.158 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:50.150+0000 7f64a8180a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:50.160 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:50.150+0000 7f64a8180a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:50.360 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-25T22:21:50.360 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:21:50.361 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:21:50.361 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:21:50.361 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:21:50.361 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:50.361 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:21:50.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:21:50.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:50.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:50.538 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:50.530+0000 7f64a8180a40 -1 Falling back to public interface 2026-02-25T22:21:50.574 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:51.218 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:51.210+0000 7f64a8180a40 -1 osd.0 6 log_to_monitors true 2026-02-25T22:21:51.575 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:51.576 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:51.576 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:21:51.577 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:21:51.577 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:51.577 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:51.782 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 6 last_clean_interval [5,6) [v2:127.0.0.1:6800/502801782,v1:127.0.0.1:6801/502801782] [v2:127.0.0.1:6802/502801782,v1:127.0.0.1:6803/502801782] exists,up ad6da4e6-32e6-476c-b5ca-ca14c7f3f5e8 2026-02-25T22:21:51.783 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:21:51.783 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:21:51.783 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:21:51.784 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: jq .osd_max_backfills 2026-02-25T22:21:51.784 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T22:21:51.784 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T22:21:51.784 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:51.784 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:51.785 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:51.785 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:51.785 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:51.785 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:51.785 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T22:21:51.785 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T22:21:51.847 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:290: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-25T22:21:51.847 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:291: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-25T22:21:51.848 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: jq .local_reservations.max_allowed 2026-02-25T22:21:51.848 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T22:21:51.849 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T22:21:51.849 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:51.849 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:51.849 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:51.849 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:51.849 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:51.850 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:51.850 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T22:21:51.850 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok dump_recovery_reservations 2026-02-25T22:21:51.910 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:295: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-25T22:21:51.910 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:296: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-25T22:21:51.912 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: jq .remote_reservations.max_allowed 2026-02-25T22:21:51.912 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: bc 2026-02-25T22:21:51.912 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T22:21:51.912 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:51.912 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:51.913 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:51.913 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:51.913 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:51.913 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:51.913 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T22:21:51.913 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok dump_recovery_reservations 2026-02-25T22:21:51.975 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:298: TEST_backfill_limit_adjustment_mclock: max_backfills=20 2026-02-25T22:21:51.975 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:299: TEST_backfill_limit_adjustment_mclock: test 20 = 20 2026-02-25T22:21:51.975 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:301: TEST_backfill_limit_adjustment_mclock: teardown td/mclock-config 2026-02-25T22:21:51.976 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:21:51.976 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:51.976 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:21:51.976 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:51.976 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:51.976 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:51.976 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:51.976 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:52.082 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:52.082 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:52.083 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:52.084 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:52.085 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:52.085 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:52.085 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:52.086 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:52.086 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:52.087 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:52.087 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:52.088 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:52.089 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:52.089 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:21:52.093 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:52.093 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:52.093 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:52.093 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:21:52.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:52.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:52.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-25T22:21:52.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:21:52.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:52.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:21:52.095 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:52.095 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:52.096 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:52.096 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:52.096 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:52.097 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:52.098 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:52.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:52.099 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:52.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:52.100 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:52.100 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:52.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:52.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:52.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:52.102 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:52.103 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:52.104 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:52.104 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:21:52.105 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:52.105 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:52.106 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:52.106 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:21:52.108 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:52.108 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:52.108 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-25T22:21:52.108 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-25T22:21:52.108 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T22:21:52.108 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T22:21:52.108 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:21:52.108 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:52.108 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:21:52.108 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:52.109 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:52.109 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:52.109 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:52.109 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:52.110 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:52.110 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:52.111 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:52.112 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:52.113 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:52.113 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:52.113 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:52.114 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:52.114 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:52.115 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:52.115 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:52.116 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:52.117 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:52.117 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:21:52.118 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:52.118 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:52.118 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:52.119 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:21:52.120 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:52.120 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:52.120 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T22:21:52.121 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:21:52.121 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:52.121 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:52.122 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11509 2026-02-25T22:21:52.123 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:21:52.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:21:52.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:21:52.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T22:21:52.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_builtin_to_custom td/mclock-config 2026-02-25T22:21:52.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:39: TEST_profile_builtin_to_custom: local dir=td/mclock-config 2026-02-25T22:21:52.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:41: TEST_profile_builtin_to_custom: run_mon td/mclock-config a 2026-02-25T22:21:52.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-25T22:21:52.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:21:52.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:21:52.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:21:52.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-25T22:21:52.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-25T22:21:52.144 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:21:52.144 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:52.144 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:52.145 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:52.145 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:52.145 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:52.145 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:52.145 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:21:52.174 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:21:52.175 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:21:52.176 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:52.176 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:52.176 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:21:52.177 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:21:52.177 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:52.177 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:52.177 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:52.177 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:52.177 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:52.177 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:52.177 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-mon.a.asok 2026-02-25T22:21:52.178 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:52.178 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11509/ceph-mon.a.asok config get fsid 2026-02-25T22:21:52.241 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:21:52.241 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:52.242 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:52.242 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:21:52.243 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:21:52.243 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:52.243 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:52.243 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:52.243 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:52.243 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:52.243 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:52.243 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-mon.a.asok 2026-02-25T22:21:52.244 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:52.244 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11509/ceph-mon.a.asok config get mon_host 2026-02-25T22:21:52.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:42: TEST_profile_builtin_to_custom: run_mgr td/mclock-config x 2026-02-25T22:21:52.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-25T22:21:52.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:21:52.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:21:52.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:21:52.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-25T22:21:52.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:21:52.417 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:21:52.417 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:52.417 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:52.418 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:52.418 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:52.418 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:52.418 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:52.418 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:52.419 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:52.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:43: TEST_profile_builtin_to_custom: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-25T22:21:52.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-25T22:21:52.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:21:52.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:21:52.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:21:52.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-25T22:21:52.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-25T22:21:52.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:21:52.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:21:52.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:21:52.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-25T22:21:52.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-25T22:21:52.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:21:52.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:21:52.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-25T22:21:52.439 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:21:52.439 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:52.440 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:52.440 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:52.440 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:52.440 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:52.440 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:52.440 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:52.440 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:21:52.440 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:21:52.440 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:21:52.440 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-25T22:21:52.440 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-25T22:21:52.440 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:21:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:21:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:21:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:21:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:21:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-25T22:21:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-25T22:21:52.441 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:21:52.442 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dfee102e-3c21-4949-a95a-65505954fe4e 2026-02-25T22:21:52.442 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 dfee102e-3c21-4949-a95a-65505954fe4e 2026-02-25T22:21:52.442 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 dfee102e-3c21-4949-a95a-65505954fe4e' 2026-02-25T22:21:52.442 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:21:52.450 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAAdp9pxJihGhAAbxHPkL0/Gljw2HsRLjcOFA== 2026-02-25T22:21:52.450 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAAdp9pxJihGhAAbxHPkL0/Gljw2HsRLjcOFA=="}' 2026-02-25T22:21:52.450 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dfee102e-3c21-4949-a95a-65505954fe4e -i td/mclock-config/0/new.json 2026-02-25T22:21:52.556 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:21:52.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-25T22:21:52.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAAdp9pxJihGhAAbxHPkL0/Gljw2HsRLjcOFA== --osd-uuid dfee102e-3c21-4949-a95a-65505954fe4e 2026-02-25T22:21:52.580 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:52.574+0000 7f0056286a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:52.582 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:52.574+0000 7f0056286a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:52.583 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:52.574+0000 7f0056286a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:52.583 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:52.574+0000 7f0056286a40 -1 bdev(0x561ee329e400 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:21:52.583 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:52.574+0000 7f0056286a40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-25T22:21:53.376 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-25T22:21:53.376 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:21:53.377 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:21:53.378 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:21:53.378 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:21:53.501 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:21:53.501 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:21:53.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-25T22:21:53.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:21:53.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:21:53.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:21:53.522 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:53.514+0000 7fe1c0db1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:53.524 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:53.518+0000 7fe1c0db1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:53.526 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:53.518+0000 7fe1c0db1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:53.613 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:21:53.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:21:53.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:21:53.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:21:53.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:21:53.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:53.614 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:21:53.615 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:21:53.615 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:53.615 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:53.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:54.006 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:53.998+0000 7fe1c0db1a40 -1 Falling back to public interface 2026-02-25T22:21:54.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:54.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:54.726 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:21:54.726 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:21:54.726 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:54.727 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:54.940 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:54.955 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:54.946+0000 7fe1c0db1a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:21:55.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:55.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:55.943 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:21:55.943 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:21:55.943 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:55.943 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:56.156 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3352107225,v1:127.0.0.1:6801/3352107225] [v2:127.0.0.1:6802/3352107225,v1:127.0.0.1:6803/3352107225] exists,up dfee102e-3c21-4949-a95a-65505954fe4e 2026-02-25T22:21:56.156 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:21:56.157 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:21:56.157 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:21:56.157 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_profile 2026-02-25T22:21:56.378 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:46: TEST_profile_builtin_to_custom: local mclock_profile=balanced 2026-02-25T22:21:56.378 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:47: TEST_profile_builtin_to_custom: test balanced = balanced 2026-02-25T22:21:56.379 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2026-02-25T22:21:56.379 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-25T22:21:56.379 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:56.380 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:56.380 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:56.381 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:56.381 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:56.381 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:56.381 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-25T22:21:56.381 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_mclock_profile 2026-02-25T22:21:56.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:52: TEST_profile_builtin_to_custom: mclock_profile='"high_recovery_ops"' 2026-02-25T22:21:56.442 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: eval echo '"high_recovery_ops"' 2026-02-25T22:21:56.442 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: echo high_recovery_ops 2026-02-25T22:21:56.442 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:53: TEST_profile_builtin_to_custom: mclock_profile=high_recovery_ops 2026-02-25T22:21:56.442 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:54: TEST_profile_builtin_to_custom: test high_recovery_ops = high_recovery_ops 2026-02-25T22:21:56.442 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:57: TEST_profile_builtin_to_custom: ceph tell osd.0 config set osd_mclock_profile custom 2026-02-25T22:21:56.538 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:21:56.538 INFO:tasks.workunit.client.0.trial188.stdout: "success": "" 2026-02-25T22:21:56.538 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:21:56.546 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: jq .osd_mclock_profile 2026-02-25T22:21:56.546 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-25T22:21:56.546 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:56.546 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:56.547 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:56.547 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:56.547 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:56.548 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:56.548 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-25T22:21:56.548 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_mclock_profile 2026-02-25T22:21:56.607 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:61: TEST_profile_builtin_to_custom: mclock_profile='"custom"' 2026-02-25T22:21:56.607 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: eval echo '"custom"' 2026-02-25T22:21:56.608 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: echo custom 2026-02-25T22:21:56.608 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:62: TEST_profile_builtin_to_custom: mclock_profile=custom 2026-02-25T22:21:56.608 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:63: TEST_profile_builtin_to_custom: test custom = custom 2026-02-25T22:21:56.608 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2026-02-25T22:21:56.608 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: bc 2026-02-25T22:21:56.609 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-25T22:21:56.609 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:56.610 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:56.610 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:56.610 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:56.610 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:56.610 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:56.610 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-25T22:21:56.611 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-25T22:21:56.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:68: TEST_profile_builtin_to_custom: local client_res=0.000000 2026-02-25T22:21:56.672 INFO:tasks.workunit.client.0.trial188.stdout:client_res = 0.000000 2026-02-25T22:21:56.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:69: TEST_profile_builtin_to_custom: echo 'client_res = 0.000000' 2026-02-25T22:21:56.673 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: echo '0.000000 + 0.1' 2026-02-25T22:21:56.673 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: bc -l 2026-02-25T22:21:56.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:70: TEST_profile_builtin_to_custom: local client_res_new=.100000 2026-02-25T22:21:56.674 INFO:tasks.workunit.client.0.trial188.stdout:client_res_new = .100000 2026-02-25T22:21:56.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:71: TEST_profile_builtin_to_custom: echo 'client_res_new = .100000' 2026-02-25T22:21:56.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:72: TEST_profile_builtin_to_custom: ceph config set osd.0 osd_mclock_scheduler_client_res .100000 2026-02-25T22:21:56.891 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-25T22:21:57.109 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:77: TEST_profile_builtin_to_custom: local res=0.100000 2026-02-25T22:21:57.110 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: echo '0.100000 != .100000' 2026-02-25T22:21:57.110 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: bc -l 2026-02-25T22:21:57.111 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:78: TEST_profile_builtin_to_custom: (( 0 )) 2026-02-25T22:21:57.112 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: jq .osd_mclock_scheduler_client_res 2026-02-25T22:21:57.112 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: bc 2026-02-25T22:21:57.113 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: get_asok_path osd.0 2026-02-25T22:21:57.113 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:21:57.113 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:21:57.114 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:57.114 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:57.114 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:57.114 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:21:57.114 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: CEPH_ARGS= 2026-02-25T22:21:57.114 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-25T22:21:57.174 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:84: TEST_profile_builtin_to_custom: res=0.100000 2026-02-25T22:21:57.175 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: echo '0.100000 != .100000' 2026-02-25T22:21:57.175 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: bc -l 2026-02-25T22:21:57.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:85: TEST_profile_builtin_to_custom: (( 0 )) 2026-02-25T22:21:57.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:89: TEST_profile_builtin_to_custom: teardown td/mclock-config 2026-02-25T22:21:57.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:21:57.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:57.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:21:57.178 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:57.178 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:57.178 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:57.178 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:57.178 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:57.284 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:57.285 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:57.285 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:57.286 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:57.287 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:57.287 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:57.288 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:57.289 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:57.289 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:57.289 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:57.289 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:57.290 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:57.292 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:57.292 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:21:57.296 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:57.297 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:57.297 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:57.297 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:21:57.299 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:57.299 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:57.299 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-25T22:21:57.299 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:21:57.299 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:57.299 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:21:57.299 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:57.299 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:57.299 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:57.299 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:57.300 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:57.301 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:57.301 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:57.302 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:57.302 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:57.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:57.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:57.303 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:57.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:57.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:57.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:57.305 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:57.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:57.307 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:57.307 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:21:57.308 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:57.308 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:57.308 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:57.309 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:21:57.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:57.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:57.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-25T22:21:57.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-25T22:21:57.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T22:21:57.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T22:21:57.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:21:57.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:57.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:21:57.311 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:57.311 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:57.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:57.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:57.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:57.313 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:57.313 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:57.314 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:57.314 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:57.315 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:57.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:57.316 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:57.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:57.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:57.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:57.317 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:57.318 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:57.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:57.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:21:57.321 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:57.321 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:57.321 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:57.321 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:21:57.322 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:57.322 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:57.323 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T22:21:57.324 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:21:57.324 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:57.324 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:57.324 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11509 2026-02-25T22:21:57.325 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:21:57.326 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:21:57.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:21:57.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T22:21:57.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_profile_custom_to_builtin td/mclock-config 2026-02-25T22:21:57.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:93: TEST_profile_custom_to_builtin: local dir=td/mclock-config 2026-02-25T22:21:57.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:95: TEST_profile_custom_to_builtin: setup td/mclock-config 2026-02-25T22:21:57.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T22:21:57.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T22:21:57.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:21:57.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:21:57.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:21:57.328 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:21:57.328 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:21:57.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:21:57.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:21:57.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:21:57.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:21:57.329 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:21:57.329 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:21:57.330 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:21:57.331 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:21:57.331 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:21:57.331 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:21:57.332 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:57.332 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:21:57.333 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:21:57.333 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:21:57.334 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:21:57.335 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:21:57.335 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:21:57.336 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:21:57.336 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:57.336 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:57.336 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:21:57.337 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:21:57.338 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:21:57.338 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T22:21:57.339 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:21:57.339 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:57.339 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:57.339 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11509 2026-02-25T22:21:57.341 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:21:57.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:21:57.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:21:57.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T22:21:57.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:96: TEST_profile_custom_to_builtin: run_mon td/mclock-config a 2026-02-25T22:21:57.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-25T22:21:57.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:21:57.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:21:57.343 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:21:57.343 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-25T22:21:57.343 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-25T22:21:57.363 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:21:57.363 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:57.363 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:57.363 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:57.363 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:57.363 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:57.364 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:57.364 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:21:57.392 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:21:57.393 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:21:57.393 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:57.393 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:57.393 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:21:57.394 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:21:57.394 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:57.394 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:57.394 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:57.394 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:57.395 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:57.395 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:57.395 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-mon.a.asok 2026-02-25T22:21:57.395 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:57.395 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11509/ceph-mon.a.asok config get fsid 2026-02-25T22:21:57.449 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:21:57.449 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:21:57.449 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:21:57.449 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:21:57.450 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:21:57.451 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:21:57.451 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:21:57.451 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:21:57.451 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:21:57.451 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:57.451 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:57.451 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-mon.a.asok 2026-02-25T22:21:57.451 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:21:57.451 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11509/ceph-mon.a.asok config get mon_host 2026-02-25T22:21:57.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:97: TEST_profile_custom_to_builtin: run_mgr td/mclock-config x 2026-02-25T22:21:57.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-25T22:21:57.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:21:57.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:21:57.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:21:57.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-25T22:21:57.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:21:57.615 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:21:57.615 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:57.616 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:57.616 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:57.616 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:57.616 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:57.616 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:57.617 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:57.617 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:21:57.637 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:98: TEST_profile_custom_to_builtin: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-25T22:21:57.638 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-25T22:21:57.638 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:21:57.638 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:21:57.638 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:21:57.638 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-25T22:21:57.638 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-25T22:21:57.638 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:21:57.638 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:21:57.638 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:21:57.638 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-25T22:21:57.639 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-25T22:21:57.639 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:21:57.639 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:21:57.639 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-25T22:21:57.639 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:21:57.639 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:21:57.639 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:21:57.639 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:21:57.639 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:21:57.639 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:21:57.639 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:57.639 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:21:57.639 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:21:57.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:21:57.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:21:57.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-25T22:21:57.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-25T22:21:57.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:21:57.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:21:57.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:21:57.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:21:57.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:21:57.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-25T22:21:57.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-25T22:21:57.641 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:21:57.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=db9e1c0f-0387-49c2-9e3c-13139b1f46d5 2026-02-25T22:21:57.641 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 db9e1c0f-0387-49c2-9e3c-13139b1f46d5 2026-02-25T22:21:57.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 db9e1c0f-0387-49c2-9e3c-13139b1f46d5' 2026-02-25T22:21:57.642 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:21:57.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAFdp9p4xKSJhAAXF091Q1Bu4jSNjwKN1P22Q== 2026-02-25T22:21:57.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAFdp9p4xKSJhAAXF091Q1Bu4jSNjwKN1P22Q=="}' 2026-02-25T22:21:57.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new db9e1c0f-0387-49c2-9e3c-13139b1f46d5 -i td/mclock-config/0/new.json 2026-02-25T22:21:57.763 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:21:57.774 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-25T22:21:57.775 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAFdp9p4xKSJhAAXF091Q1Bu4jSNjwKN1P22Q== --osd-uuid db9e1c0f-0387-49c2-9e3c-13139b1f46d5 2026-02-25T22:21:57.790 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:57.782+0000 7f16542eaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:57.792 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:57.786+0000 7f16542eaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:57.793 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:57.786+0000 7f16542eaa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:57.794 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:57.786+0000 7f16542eaa40 -1 bdev(0x562ccde98400 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:21:57.794 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:57.786+0000 7f16542eaa40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-25T22:21:58.515 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-25T22:21:58.515 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:21:58.516 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:21:58.516 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:21:58.517 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:21:58.626 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:21:58.627 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:21:58.627 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-25T22:21:58.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:21:58.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:21:58.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:21:58.650 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:58.642+0000 7fbed0c06a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:58.652 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:58.646+0000 7fbed0c06a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:58.654 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:58.646+0000 7fbed0c06a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:21:58.733 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:21:58.734 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:21:58.734 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:21:58.734 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:21:58.734 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:21:58.734 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:58.734 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:21:58.734 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:21:58.735 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:58.735 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:58.838 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:21:59.057 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:59.050+0000 7fbed0c06a40 -1 Falling back to public interface 2026-02-25T22:21:59.746 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:21:59.738+0000 7fbed0c06a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:21:59.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:21:59.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:21:59.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:21:59.840 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:21:59.841 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:21:59.841 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:21:59.956 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:00.958 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:00.958 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:00.958 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:22:00.959 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:00.959 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:00.959 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:01.175 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3351186267,v1:127.0.0.1:6801/3351186267] [v2:127.0.0.1:6802/3351186267,v1:127.0.0.1:6803/3351186267] exists,up db9e1c0f-0387-49c2-9e3c-13139b1f46d5 2026-02-25T22:22:01.176 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:01.176 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:01.176 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:01.176 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:101: TEST_profile_custom_to_builtin: local def_mclock_profile 2026-02-25T22:22:01.176 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_profile 2026-02-25T22:22:01.390 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:102: TEST_profile_custom_to_builtin: def_mclock_profile=balanced 2026-02-25T22:22:01.390 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:103: TEST_profile_custom_to_builtin: test balanced = balanced 2026-02-25T22:22:01.390 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-25T22:22:01.391 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T22:22:01.391 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:01.391 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:01.392 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:01.392 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:01.392 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:01.392 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:22:01.392 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T22:22:01.392 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_mclock_profile 2026-02-25T22:22:01.453 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:108: TEST_profile_custom_to_builtin: local 'orig_mclock_profile="high_recovery_ops"' 2026-02-25T22:22:01.454 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: eval echo '"high_recovery_ops"' 2026-02-25T22:22:01.454 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: echo high_recovery_ops 2026-02-25T22:22:01.454 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:109: TEST_profile_custom_to_builtin: orig_mclock_profile=high_recovery_ops 2026-02-25T22:22:01.454 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:110: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2026-02-25T22:22:01.454 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:113: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile custom 2026-02-25T22:22:01.546 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:22:01.546 INFO:tasks.workunit.client.0.trial188.stdout: "success": "" 2026-02-25T22:22:01.546 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:22:01.554 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-25T22:22:01.554 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T22:22:01.554 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:01.554 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:01.555 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:01.555 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:01.555 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:01.556 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:22:01.556 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T22:22:01.556 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_mclock_profile 2026-02-25T22:22:01.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:118: TEST_profile_custom_to_builtin: local 'mclock_profile="custom"' 2026-02-25T22:22:01.615 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: eval echo '"custom"' 2026-02-25T22:22:01.615 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: echo custom 2026-02-25T22:22:01.615 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:119: TEST_profile_custom_to_builtin: mclock_profile=custom 2026-02-25T22:22:01.616 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:120: TEST_profile_custom_to_builtin: test custom = custom 2026-02-25T22:22:01.616 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:123: TEST_profile_custom_to_builtin: local client_res 2026-02-25T22:22:01.617 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-25T22:22:01.617 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: bc 2026-02-25T22:22:01.617 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T22:22:01.617 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:01.617 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:01.617 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:01.617 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:01.617 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:01.618 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:22:01.618 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T22:22:01.618 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-25T22:22:01.679 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:126: TEST_profile_custom_to_builtin: client_res=0.000000 2026-02-25T22:22:01.679 INFO:tasks.workunit.client.0.trial188.stdout:Original client_res for osd.0 = 0.000000 2026-02-25T22:22:01.680 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:127: TEST_profile_custom_to_builtin: echo 'Original client_res for osd.0 = 0.000000' 2026-02-25T22:22:01.680 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: echo '0.000000 + 0.1' 2026-02-25T22:22:01.680 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: bc -l 2026-02-25T22:22:01.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:130: TEST_profile_custom_to_builtin: local client_res_new=.100000 2026-02-25T22:22:01.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:131: TEST_profile_custom_to_builtin: echo 'client_res_new = .100000' 2026-02-25T22:22:01.682 INFO:tasks.workunit.client.0.trial188.stdout:client_res_new = .100000 2026-02-25T22:22:01.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:132: TEST_profile_custom_to_builtin: ceph config set osd osd_mclock_scheduler_client_res .100000 2026-02-25T22:22:01.902 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-25T22:22:02.134 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:136: TEST_profile_custom_to_builtin: local res=0.100000 2026-02-25T22:22:02.134 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: echo '0.100000 != .100000' 2026-02-25T22:22:02.135 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: bc -l 2026-02-25T22:22:02.136 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:137: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-25T22:22:02.137 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-25T22:22:02.137 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: bc 2026-02-25T22:22:02.137 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T22:22:02.137 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:02.137 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:02.138 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:02.138 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:02.138 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:02.139 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:22:02.139 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T22:22:02.139 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-25T22:22:02.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:143: TEST_profile_custom_to_builtin: res=0.100000 2026-02-25T22:22:02.199 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: echo '0.100000 != .100000' 2026-02-25T22:22:02.199 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: bc -l 2026-02-25T22:22:02.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:144: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-25T22:22:02.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:154: TEST_profile_custom_to_builtin: ceph tell osd.0 config set osd_mclock_profile high_recovery_ops 2026-02-25T22:22:02.266 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:22:02.266 INFO:tasks.workunit.client.0.trial188.stdout: "success": "" 2026-02-25T22:22:02.266 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:22:02.274 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: jq .osd_mclock_profile 2026-02-25T22:22:02.275 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T22:22:02.275 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:02.275 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:02.275 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:02.275 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:02.275 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:02.275 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:22:02.276 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T22:22:02.276 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_mclock_profile 2026-02-25T22:22:02.338 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: eval 'mclock_profile="high_recovery_ops"' 2026-02-25T22:22:02.338 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:158: TEST_profile_custom_to_builtin: mclock_profile=high_recovery_ops 2026-02-25T22:22:02.338 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:160: TEST_profile_custom_to_builtin: test high_recovery_ops = high_recovery_ops 2026-02-25T22:22:02.339 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-25T22:22:02.546 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:165: TEST_profile_custom_to_builtin: local res=0.100000 2026-02-25T22:22:02.547 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: echo '0.100000 != .100000' 2026-02-25T22:22:02.547 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: bc -l 2026-02-25T22:22:02.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:166: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-25T22:22:02.549 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-25T22:22:02.550 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: bc 2026-02-25T22:22:02.550 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T22:22:02.550 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:02.550 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:02.551 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:02.551 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:02.551 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:02.551 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:22:02.551 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T22:22:02.552 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-25T22:22:02.609 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:172: TEST_profile_custom_to_builtin: res=0.100000 2026-02-25T22:22:02.610 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: echo '0.100000 != .100000' 2026-02-25T22:22:02.610 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: bc -l 2026-02-25T22:22:02.611 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:173: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-25T22:22:02.611 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:178: TEST_profile_custom_to_builtin: ceph config rm osd osd_mclock_scheduler_client_res 2026-02-25T22:22:02.825 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:180: TEST_profile_custom_to_builtin: sleep 5 2026-02-25T22:22:07.827 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: ceph config get osd.0 osd_mclock_scheduler_client_res 2026-02-25T22:22:08.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:185: TEST_profile_custom_to_builtin: res=0.000000 2026-02-25T22:22:08.035 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: echo '0.000000 != 0.0' 2026-02-25T22:22:08.036 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: bc -l 2026-02-25T22:22:08.037 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:186: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-25T22:22:08.038 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: jq .osd_mclock_scheduler_client_res 2026-02-25T22:22:08.038 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: bc 2026-02-25T22:22:08.038 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: get_asok_path osd.0 2026-02-25T22:22:08.038 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:08.038 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:08.039 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:08.039 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.039 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:08.039 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:22:08.039 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: CEPH_ARGS= 2026-02-25T22:22:08.040 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_mclock_scheduler_client_res 2026-02-25T22:22:08.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:193: TEST_profile_custom_to_builtin: res=0.000000 2026-02-25T22:22:08.099 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: echo '0.000000 != 0.000000' 2026-02-25T22:22:08.099 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: bc -l 2026-02-25T22:22:08.100 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:194: TEST_profile_custom_to_builtin: (( 0 )) 2026-02-25T22:22:08.100 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:198: TEST_profile_custom_to_builtin: teardown td/mclock-config 2026-02-25T22:22:08.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:22:08.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:08.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:22:08.101 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:08.101 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:08.102 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:08.102 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:08.102 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:08.208 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:08.209 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:08.210 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:08.210 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:08.211 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:08.211 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:08.212 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:08.213 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:08.213 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:08.213 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:08.213 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:08.214 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:08.215 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:08.215 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:22:08.220 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:08.220 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.220 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:08.220 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:22:08.222 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:08.222 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:08.222 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-25T22:22:08.222 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:22:08.222 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:08.222 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:22:08.223 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:08.223 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:08.223 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:08.223 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:08.223 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:08.225 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:08.225 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:08.226 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:08.226 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:08.227 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:08.227 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:08.228 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:08.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:08.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:08.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:08.229 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:08.230 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:08.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:08.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:22:08.233 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:08.233 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.233 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:08.234 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:22:08.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:08.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:08.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:31: run: for func in $funcs 2026-02-25T22:22:08.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:32: run: setup td/mclock-config 2026-02-25T22:22:08.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T22:22:08.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T22:22:08.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:22:08.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:08.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:22:08.236 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:08.237 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:08.237 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:08.237 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:08.237 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:08.238 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:08.238 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:08.239 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:08.240 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:08.240 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:08.241 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:08.241 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:08.242 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:08.242 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:08.243 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:08.243 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:08.244 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:08.245 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:08.245 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:22:08.246 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:08.246 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.246 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:08.246 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:22:08.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:08.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:08.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T22:22:08.249 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:22:08.249 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.249 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:08.249 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11509 2026-02-25T22:22:08.250 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:22:08.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:22:08.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:22:08.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T22:22:08.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:33: run: TEST_recovery_limit_adjustment_mclock td/mclock-config 2026-02-25T22:22:08.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:202: TEST_recovery_limit_adjustment_mclock: local dir=td/mclock-config 2026-02-25T22:22:08.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:204: TEST_recovery_limit_adjustment_mclock: setup td/mclock-config 2026-02-25T22:22:08.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mclock-config 2026-02-25T22:22:08.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mclock-config 2026-02-25T22:22:08.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:22:08.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:08.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:22:08.253 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:08.253 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:08.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:08.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:08.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:08.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:08.254 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:08.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:08.255 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:08.257 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:08.257 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:08.257 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:08.258 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:08.258 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:08.259 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:08.259 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:08.260 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:08.261 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:08.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:22:08.262 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:08.263 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.263 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:08.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:22:08.264 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:08.264 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:08.264 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mclock-config 2026-02-25T22:22:08.267 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:22:08.268 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.269 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:08.269 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.11509 2026-02-25T22:22:08.269 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:22:08.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:22:08.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:22:08.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mclock-config 1' TERM HUP INT 2026-02-25T22:22:08.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:205: TEST_recovery_limit_adjustment_mclock: run_mon td/mclock-config a 2026-02-25T22:22:08.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mclock-config 2026-02-25T22:22:08.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:22:08.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:22:08.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:22:08.272 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mclock-config/a 2026-02-25T22:22:08.272 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mclock-config/a --run-dir=td/mclock-config 2026-02-25T22:22:08.289 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:22:08.289 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:08.289 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:08.289 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:08.290 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.290 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:08.290 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:22:08.290 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mclock-config/a '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --mon-cluster-log-file=td/mclock-config/log --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:22:08.319 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:22:08.320 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:22:08.320 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:08.320 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:08.320 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:22:08.322 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:22:08.322 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:08.322 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:08.322 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:08.322 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:08.322 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.322 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:08.322 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-mon.a.asok 2026-02-25T22:22:08.323 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:08.323 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11509/ceph-mon.a.asok config get fsid 2026-02-25T22:22:08.376 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:22:08.376 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:08.376 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:08.376 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:22:08.376 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:22:08.376 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:08.377 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:08.377 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:08.378 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:08.378 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.378 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:08.378 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-mon.a.asok 2026-02-25T22:22:08.378 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:08.378 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.11509/ceph-mon.a.asok config get mon_host 2026-02-25T22:22:08.430 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:206: TEST_recovery_limit_adjustment_mclock: run_mgr td/mclock-config x 2026-02-25T22:22:08.431 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mclock-config 2026-02-25T22:22:08.431 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:22:08.431 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:22:08.431 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:22:08.431 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mclock-config/x 2026-02-25T22:22:08.431 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:22:08.539 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:22:08.539 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:08.539 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:08.539 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:08.539 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.539 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:08.540 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:22:08.540 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:08.540 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mclock-config/x '--log-file=td/mclock-config/$name.log' '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --run-dir=td/mclock-config '--pid-file=td/mclock-config/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:08.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:208: TEST_recovery_limit_adjustment_mclock: run_osd td/mclock-config 0 --osd_op_queue=mclock_scheduler 2026-02-25T22:22:08.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mclock-config 2026-02-25T22:22:08.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:08.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:22:08.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:08.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mclock-config/0 2026-02-25T22:22:08.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 ' 2026-02-25T22:22:08.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:08.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:08.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:08.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mclock-config/0' 2026-02-25T22:22:08.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mclock-config/0/journal' 2026-02-25T22:22:08.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:08.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:08.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mclock-config' 2026-02-25T22:22:08.563 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mclock-config/$name.log' 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mclock-config/$name.pid' 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:08.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:08.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:08.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:08.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:08.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-25T22:22:08.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mclock-config/0 2026-02-25T22:22:08.565 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=42de8c5e-b0f0-4ae5-9257-c0120ef2a1bc 2026-02-25T22:22:08.566 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 42de8c5e-b0f0-4ae5-9257-c0120ef2a1bc 2026-02-25T22:22:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 42de8c5e-b0f0-4ae5-9257-c0120ef2a1bc' 2026-02-25T22:22:08.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:08.574 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAQdp9p5vQBIhAAVlzwniGF+/X/5ceOTdCE6g== 2026-02-25T22:22:08.574 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAQdp9p5vQBIhAAVlzwniGF+/X/5ceOTdCE6g=="}' 2026-02-25T22:22:08.574 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 42de8c5e-b0f0-4ae5-9257-c0120ef2a1bc -i td/mclock-config/0/new.json 2026-02-25T22:22:08.692 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:22:08.699 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mclock-config/0/new.json 2026-02-25T22:22:08.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQAQdp9p5vQBIhAAVlzwniGF+/X/5ceOTdCE6g== --osd-uuid 42de8c5e-b0f0-4ae5-9257-c0120ef2a1bc 2026-02-25T22:22:08.716 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:08.706+0000 7fb764718a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:08.717 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:08.710+0000 7fb764718a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:08.718 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:08.710+0000 7fb764718a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:08.718 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:08.710+0000 7fb764718a40 -1 bdev(0x5634c2b40400 td/mclock-config/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:08.718 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:08.710+0000 7fb764718a40 -1 bluestore(td/mclock-config/0) _read_fsid unparsable uuid 2026-02-25T22:22:09.384 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mclock-config/0/keyring 2026-02-25T22:22:09.384 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:09.385 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:22:09.385 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:22:09.385 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mclock-config/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:09.496 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:22:09.497 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:22:09.497 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=f34ec4b0-2802-4b6f-92cc-ca37b506196d --auth-supported=none --mon-host=127.0.0.1:7124 --debug-mclock 20 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mclock-config/0 --osd-journal=td/mclock-config/0/journal --chdir= --run-dir=td/mclock-config '--admin-socket=/tmp/ceph-asok.11509/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mclock-config/$name.log' '--pid-file=td/mclock-config/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-25T22:22:09.497 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:09.497 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:09.497 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:09.510 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:09.502+0000 7f48dcb34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:09.511 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:09.502+0000 7f48dcb34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:09.512 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:09.506+0000 7f48dcb34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:09.610 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:22:09.610 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:09.610 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:22:09.610 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:09.610 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:09.610 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:09.611 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:22:09.611 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:09.611 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:09.612 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:09.719 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:09.918 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:09.910+0000 7f48dcb34a40 -1 Falling back to public interface 2026-02-25T22:22:10.721 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:10.721 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:10.721 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:10.721 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:22:10.723 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:10.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:10.845 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:10.979 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:10.970+0000 7f48dcb34a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:22:11.846 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:11.846 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:11.847 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:22:11.847 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:11.848 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:11.848 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:12.063 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1951972223,v1:127.0.0.1:6801/1951972223] [v2:127.0.0.1:6802/1951972223,v1:127.0.0.1:6803/1951972223] exists,up 42de8c5e-b0f0-4ae5-9257-c0120ef2a1bc 2026-02-25T22:22:12.064 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:12.064 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:12.064 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:12.064 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T22:22:12.064 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:12.064 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:12.065 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:12.065 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:12.065 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:12.065 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:22:12.065 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T22:22:12.065 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-25T22:22:12.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:210: TEST_recovery_limit_adjustment_mclock: local 'recoveries={"osd_recovery_max_active":"0"}' 2026-02-25T22:22:12.126 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: echo '{"osd_recovery_max_active":"0"}' 2026-02-25T22:22:12.126 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:212: TEST_recovery_limit_adjustment_mclock: grep --quiet osd_recovery_max_active 2026-02-25T22:22:12.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:217: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2026-02-25T22:22:12.350 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:218: TEST_recovery_limit_adjustment_mclock: sleep 2 2026-02-25T22:22:14.352 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T22:22:14.353 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:14.353 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:14.354 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:14.354 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:14.354 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:14.354 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:22:14.354 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T22:22:14.354 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-25T22:22:14.412 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:220: TEST_recovery_limit_adjustment_mclock: local 'max_recoveries={"osd_recovery_max_active":"0"}' 2026-02-25T22:22:14.413 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:221: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"0"}' = '{"osd_recovery_max_active":"0"}' 2026-02-25T22:22:14.413 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:225: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_mclock_override_recovery_settings true 2026-02-25T22:22:14.632 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:226: TEST_recovery_limit_adjustment_mclock: ceph config set osd.0 osd_recovery_max_active 10 2026-02-25T22:22:14.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:227: TEST_recovery_limit_adjustment_mclock: sleep 2 2026-02-25T22:22:16.868 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: get_asok_path osd.0 2026-02-25T22:22:16.869 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:16.869 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:16.870 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:16.870 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:16.870 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:16.870 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.11509/ceph-osd.0.asok 2026-02-25T22:22:16.870 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: CEPH_ARGS= 2026-02-25T22:22:16.870 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: ceph --format=json daemon /tmp/ceph-asok.11509/ceph-osd.0.asok config get osd_recovery_max_active 2026-02-25T22:22:16.931 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:229: TEST_recovery_limit_adjustment_mclock: max_recoveries='{"osd_recovery_max_active":"10"}' 2026-02-25T22:22:16.931 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:230: TEST_recovery_limit_adjustment_mclock: test '{"osd_recovery_max_active":"10"}' = '{"osd_recovery_max_active":"10"}' 2026-02-25T22:22:16.931 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:232: TEST_recovery_limit_adjustment_mclock: teardown td/mclock-config 2026-02-25T22:22:16.931 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:22:16.931 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:16.931 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:22:16.931 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:16.931 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:16.932 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:16.932 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:16.932 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:17.039 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:17.039 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:17.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:17.040 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:17.042 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:17.042 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:17.042 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:17.043 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:17.044 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:17.044 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:17.044 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:17.045 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:17.046 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:17.046 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:22:17.051 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:17.051 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.051 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:17.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:22:17.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:17.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:17.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/mclock-config.sh:34: run: teardown td/mclock-config 2026-02-25T22:22:17.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:22:17.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:17.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:22:17.054 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:17.054 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:17.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:17.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:17.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:17.056 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:17.056 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:17.057 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:17.057 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:17.058 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:17.058 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:17.059 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:17.060 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:17.060 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:17.060 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:17.061 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:17.062 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:17.063 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:17.063 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:22:17.064 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:17.064 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.064 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:17.064 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:22:17.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:17.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:17.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T22:22:17.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/mclock-config 0 2026-02-25T22:22:17.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mclock-config 2026-02-25T22:22:17.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T22:22:17.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mclock-config KILL 2026-02-25T22:22:17.067 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:17.067 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:17.067 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:17.067 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:17.067 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:17.068 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:17.069 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:17.069 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:17.070 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:17.071 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:17.071 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:17.071 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:17.072 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:17.072 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:17.073 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:17.073 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:17.074 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:17.075 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T22:22:17.075 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mclock-config 2026-02-25T22:22:17.076 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:17.077 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.077 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.11509 2026-02-25T22:22:17.077 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.11509 2026-02-25T22:22:17.078 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:17.078 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:17.078 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T22:22:17.079 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T22:22:17.080 DEBUG:teuthology.orchestra.run.trial188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T22:22:17.129 INFO:tasks.workunit:Running workunit misc/network-ping.sh... 2026-02-25T22:22:17.129 DEBUG:teuthology.orchestra.run.trial188:workunit test misc/network-ping.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61d4431d3ec879bd98659388b3655acf3d392418 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh 2026-02-25T22:22:17.181 INFO:tasks.workunit.client.0.trial188.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T22:22:17.184 INFO:tasks.workunit.client.0.trial188.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T22:22:17.184 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:22:17.184 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:22:17.184 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T22:22:17.184 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T22:22:17.185 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T22:22:17.185 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T22:22:17.185 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T22:22:17.185 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T22:22:17.185 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/network-ping 2026-02-25T22:22:17.185 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:6: run: local dir=td/network-ping 2026-02-25T22:22:17.185 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:7: run: shift 2026-02-25T22:22:17.185 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: export CEPH_MON=127.0.0.1:7146 2026-02-25T22:22:17.185 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:9: run: CEPH_MON=127.0.0.1:7146 2026-02-25T22:22:17.185 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:10: run: export CEPH_ARGS 2026-02-25T22:22:17.185 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: uuidgen 2026-02-25T22:22:17.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:11: run: CEPH_ARGS+='--fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none ' 2026-02-25T22:22:17.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:12: run: CEPH_ARGS+='--mon-host=127.0.0.1:7146 ' 2026-02-25T22:22:17.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:13: run: CEPH_ARGS+='--debug_disable_randomized_ping=true ' 2026-02-25T22:22:17.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:14: run: CEPH_ARGS+='--debug_heartbeat_testing_span=5 ' 2026-02-25T22:22:17.187 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:15: run: CEPH_ARGS+='--osd_heartbeat_interval=1 ' 2026-02-25T22:22:17.187 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: set 2026-02-25T22:22:17.187 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-25T22:22:17.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:16: run: local 'funcs=TEST_network_ping_test1 2026-02-25T22:22:17.189 INFO:tasks.workunit.client.0.trial188.stderr:TEST_network_ping_test2' 2026-02-25T22:22:17.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2026-02-25T22:22:17.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2026-02-25T22:22:17.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/network-ping 2026-02-25T22:22:17.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/network-ping 2026-02-25T22:22:17.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-25T22:22:17.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:22:17.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-25T22:22:17.189 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:22:17.189 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:22:17.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:22:17.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:22:17.190 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:22:17.190 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:22:17.190 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:22:17.191 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:22:17.191 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:22:17.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:22:17.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:22:17.193 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:22:17.193 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:17.193 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:22:17.194 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:22:17.194 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:22:17.195 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:22:17.196 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:22:17.196 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-25T22:22:17.197 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:22:17.197 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.197 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:22:17.197 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.24105 2026-02-25T22:22:17.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:22:17.199 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:22:17.199 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/network-ping 2026-02-25T22:22:17.200 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:22:17.200 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.200 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:22:17.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.24105 2026-02-25T22:22:17.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:22:17.203 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T22:22:17.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T22:22:17.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:22:17.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2026-02-25T22:22:17.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test1 td/network-ping 2026-02-25T22:22:17.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:25: TEST_network_ping_test1: local dir=td/network-ping 2026-02-25T22:22:17.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:27: TEST_network_ping_test1: run_mon td/network-ping a 2026-02-25T22:22:17.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/network-ping 2026-02-25T22:22:17.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:22:17.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:22:17.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:22:17.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/network-ping/a 2026-02-25T22:22:17.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2026-02-25T22:22:17.227 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:22:17.228 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:17.228 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:17.228 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:17.228 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.228 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:22:17.228 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:22:17.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:22:17.258 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:22:17.259 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:22:17.259 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:17.259 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:17.259 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:22:17.260 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:22:17.261 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:17.261 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:17.261 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:17.261 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:17.261 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.261 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:22:17.261 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-mon.a.asok 2026-02-25T22:22:17.261 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:17.261 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.24105/ceph-mon.a.asok config get fsid 2026-02-25T22:22:17.317 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:22:17.317 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:22:17.317 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:22:17.317 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:22:17.318 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:22:17.319 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:22:17.319 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:22:17.319 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:22:17.319 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:17.319 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.319 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:22:17.319 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-mon.a.asok 2026-02-25T22:22:17.319 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:22:17.319 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.24105/ceph-mon.a.asok config get mon_host 2026-02-25T22:22:17.375 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:28: TEST_network_ping_test1: run_mgr td/network-ping x 2026-02-25T22:22:17.375 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/network-ping 2026-02-25T22:22:17.375 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:22:17.376 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:22:17.376 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:22:17.376 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/network-ping/x 2026-02-25T22:22:17.376 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:22:17.488 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:22:17.488 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:17.488 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:17.488 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:17.489 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.489 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:22:17.489 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:22:17.489 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:17.490 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:22:17.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:29: TEST_network_ping_test1: run_osd td/network-ping 0 2026-02-25T22:22:17.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-25T22:22:17.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:17.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:22:17.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:17.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/0 2026-02-25T22:22:17.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-25T22:22:17.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:17.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:17.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:17.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2026-02-25T22:22:17.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2026-02-25T22:22:17.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:17.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:17.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-25T22:22:17.511 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:17.511 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:17.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:17.513 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:17.513 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:17.513 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:22:17.513 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/0 2026-02-25T22:22:17.513 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:17.514 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=11c5a49f-d66c-4338-943f-682939679d01 2026-02-25T22:22:17.514 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 11c5a49f-d66c-4338-943f-682939679d01' 2026-02-25T22:22:17.514 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 11c5a49f-d66c-4338-943f-682939679d01 2026-02-25T22:22:17.514 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:17.523 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAZdp9pIH74HhAAwaNDj9dsLRPeYmzgxKYpQA== 2026-02-25T22:22:17.523 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAZdp9pIH74HhAAwaNDj9dsLRPeYmzgxKYpQA=="}' 2026-02-25T22:22:17.523 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 11c5a49f-d66c-4338-943f-682939679d01 -i td/network-ping/0/new.json 2026-02-25T22:22:17.632 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:22:17.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/0/new.json 2026-02-25T22:22:17.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAZdp9pIH74HhAAwaNDj9dsLRPeYmzgxKYpQA== --osd-uuid 11c5a49f-d66c-4338-943f-682939679d01 2026-02-25T22:22:17.657 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:17.650+0000 7f57f30bda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:17.659 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:17.650+0000 7f57f30bda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:17.660 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:17.650+0000 7f57f30bda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:17.660 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:17.650+0000 7f57f30bda40 -1 bdev(0x55f8f5338400 td/network-ping/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:17.660 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:17.650+0000 7f57f30bda40 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2026-02-25T22:22:18.345 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/0/keyring 2026-02-25T22:22:18.345 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:18.346 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:22:18.347 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:22:18.347 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:18.457 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:22:18.458 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:22:18.458 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:22:18.459 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:18.459 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:18.459 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:18.473 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:18.466+0000 7f75297cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:18.475 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:18.466+0000 7f75297cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:18.476 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:18.470+0000 7f75297cfa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:18.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:22:18.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:18.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:22:18.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:18.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:18.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:18.567 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:22:18.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:18.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:18.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:18.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:18.994 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:18.986+0000 7f75297cfa40 -1 Falling back to public interface 2026-02-25T22:22:19.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:19.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:19.674 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:22:19.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:19.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:19.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:19.807 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:19.973 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:19.966+0000 7f75297cfa40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:22:20.808 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:20.808 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:20.809 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:22:20.809 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:20.810 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:20.810 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:22:21.028 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1823973488,v1:127.0.0.1:6801/1823973488] [v2:127.0.0.1:6802/1823973488,v1:127.0.0.1:6803/1823973488] exists,up 11c5a49f-d66c-4338-943f-682939679d01 2026-02-25T22:22:21.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:21.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:21.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:21.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:30: TEST_network_ping_test1: run_osd td/network-ping 1 2026-02-25T22:22:21.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-25T22:22:21.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:21.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:22:21.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:21.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/1 2026-02-25T22:22:21.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-25T22:22:21.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:21.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:21.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:21.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2026-02-25T22:22:21.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2026-02-25T22:22:21.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:21.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:21.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-25T22:22:21.032 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:21.033 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:21.033 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:21.033 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:21.033 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:21.033 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:22:21.033 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:22:21.033 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:22:21.033 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:21.033 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:21.033 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:21.033 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-25T22:22:21.033 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-25T22:22:21.033 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:21.034 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:21.034 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:21.034 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:21.034 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:21.034 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:22:21.034 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/1 2026-02-25T22:22:21.034 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:21.034 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dbdd67bf-b872-4a30-a4a4-d3721783b5d1 2026-02-25T22:22:21.035 INFO:tasks.workunit.client.0.trial188.stdout:add osd1 dbdd67bf-b872-4a30-a4a4-d3721783b5d1 2026-02-25T22:22:21.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 dbdd67bf-b872-4a30-a4a4-d3721783b5d1' 2026-02-25T22:22:21.035 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:21.044 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAddp9p1N5jAhAAl5Kfq8/LGZ8lYs0d2JcS/A== 2026-02-25T22:22:21.044 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAddp9p1N5jAhAAl5Kfq8/LGZ8lYs0d2JcS/A=="}' 2026-02-25T22:22:21.044 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dbdd67bf-b872-4a30-a4a4-d3721783b5d1 -i td/network-ping/1/new.json 2026-02-25T22:22:21.270 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:22:21.279 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/1/new.json 2026-02-25T22:22:21.280 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAddp9p1N5jAhAAl5Kfq8/LGZ8lYs0d2JcS/A== --osd-uuid dbdd67bf-b872-4a30-a4a4-d3721783b5d1 2026-02-25T22:22:21.297 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:21.290+0000 7fd379325a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:21.299 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:21.290+0000 7fd379325a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:21.300 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:21.290+0000 7fd379325a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:21.300 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:21.290+0000 7fd379325a40 -1 bdev(0x560177854400 td/network-ping/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:21.300 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:21.290+0000 7fd379325a40 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2026-02-25T22:22:21.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/1/keyring 2026-02-25T22:22:21.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:21.972 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:22:21.972 INFO:tasks.workunit.client.0.trial188.stdout:adding osd1 key to auth repository 2026-02-25T22:22:21.972 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:22.272 INFO:tasks.workunit.client.0.trial188.stdout:start osd.1 2026-02-25T22:22:22.273 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:22:22.273 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:22:22.273 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:22.273 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:22.273 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:22.296 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:22.290+0000 7fb171309a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:22.298 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:22.290+0000 7fb171309a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:22.299 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:22.290+0000 7fb171309a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:22.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:22:22.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:22.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:22:22.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:22.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:22.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:22.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:22.493 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:22:22.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:22.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:22:22.706 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:22.782 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:22.774+0000 7fb171309a40 -1 Falling back to public interface 2026-02-25T22:22:23.410 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:23.402+0000 7fb171309a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:22:23.707 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:23.707 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:23.707 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:22:23.708 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:23.708 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:23.708 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:22:23.916 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:24.917 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:24.917 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:24.918 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:22:24.918 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:22:24.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:24.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:22:25.133 INFO:tasks.workunit.client.0.trial188.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1840201265,v1:127.0.0.1:6811/1840201265] [v2:127.0.0.1:6812/1840201265,v1:127.0.0.1:6813/1840201265] exists,up dbdd67bf-b872-4a30-a4a4-d3721783b5d1 2026-02-25T22:22:25.136 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:25.136 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:25.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:25.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:31: TEST_network_ping_test1: run_osd td/network-ping 2 2026-02-25T22:22:25.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-25T22:22:25.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:22:25.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:22:25.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:22:25.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/2 2026-02-25T22:22:25.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-25T22:22:25.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:22:25.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:22:25.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:22:25.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2026-02-25T22:22:25.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2026-02-25T22:22:25.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:22:25.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:22:25.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-25T22:22:25.138 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:22:25.138 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:22:25.138 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:22:25.138 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:22:25.138 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:25.138 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:22:25.138 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:22:25.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:22:25.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:22:25.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:22:25.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:22:25.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-25T22:22:25.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-25T22:22:25.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:22:25.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:22:25.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:22:25.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:22:25.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:22:25.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:22:25.139 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/2 2026-02-25T22:22:25.140 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:22:25.140 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7e892191-bf9f-4425-8ff5-b4a0e16bbe37 2026-02-25T22:22:25.140 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 7e892191-bf9f-4425-8ff5-b4a0e16bbe37' 2026-02-25T22:22:25.141 INFO:tasks.workunit.client.0.trial188.stdout:add osd2 7e892191-bf9f-4425-8ff5-b4a0e16bbe37 2026-02-25T22:22:25.141 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:22:25.151 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAhdp9p5QbKCBAAKSZ54F1zrBncYANiJeNEkw== 2026-02-25T22:22:25.151 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAhdp9p5QbKCBAAKSZ54F1zrBncYANiJeNEkw=="}' 2026-02-25T22:22:25.151 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7e892191-bf9f-4425-8ff5-b4a0e16bbe37 -i td/network-ping/2/new.json 2026-02-25T22:22:25.374 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:22:25.383 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/2/new.json 2026-02-25T22:22:25.384 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAhdp9p5QbKCBAAKSZ54F1zrBncYANiJeNEkw== --osd-uuid 7e892191-bf9f-4425-8ff5-b4a0e16bbe37 2026-02-25T22:22:25.400 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:25.390+0000 7fd9d67b3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:25.401 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:25.394+0000 7fd9d67b3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:25.402 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:25.394+0000 7fd9d67b3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:25.403 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:25.394+0000 7fd9d67b3a40 -1 bdev(0x560098946400 td/network-ping/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:22:25.403 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:25.394+0000 7fd9d67b3a40 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2026-02-25T22:22:26.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/2/keyring 2026-02-25T22:22:26.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:22:26.060 INFO:tasks.workunit.client.0.trial188.stdout:adding osd2 key to auth repository 2026-02-25T22:22:26.060 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:22:26.061 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:22:26.341 INFO:tasks.workunit.client.0.trial188.stdout:start osd.2 2026-02-25T22:22:26.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:22:26.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:22:26.343 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:22:26.343 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:22:26.343 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:22:26.358 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:26.350+0000 7fa99a379a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:26.360 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:26.350+0000 7fa99a379a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:26.361 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:26.354+0000 7fa99a379a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:22:26.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:22:26.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:22:26.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:22:26.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:22:26.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:22:26.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:26.566 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:22:26.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:22:26.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:26.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:22:26.774 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:26.765+0000 7fa99a379a40 -1 Falling back to public interface 2026-02-25T22:22:26.782 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:22:27.378 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:22:27.369+0000 7fa99a379a40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:22:27.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:22:27.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:22:27.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:22:27.784 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:22:27.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:22:27.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:22:28.010 INFO:tasks.workunit.client.0.trial188.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/3741466211,v1:127.0.0.1:6819/3741466211] [v2:127.0.0.1:6820/3741466211,v1:127.0.0.1:6821/3741466211] exists,up 7e892191-bf9f-4425-8ff5-b4a0e16bbe37 2026-02-25T22:22:28.010 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:22:28.010 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:22:28.011 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:22:28.011 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:33: TEST_network_ping_test1: sleep 5 2026-02-25T22:22:33.011 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:35: TEST_network_ping_test1: create_pool foo 16 2026-02-25T22:22:33.012 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 2026-02-25T22:22:33.276 INFO:tasks.workunit.client.0.trial188.stderr:pool 'foo' already exists 2026-02-25T22:22:33.286 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:22:34.287 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:38: TEST_network_ping_test1: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2026-02-25T22:22:34.308 INFO:tasks.workunit.client.0.trial188.stdout:hints = 1 2026-02-25T22:22:34.309 INFO:tasks.workunit.client.0.trial188.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2026-02-25T22:22:34.309 INFO:tasks.workunit.client.0.trial188.stdout:Object prefix: benchmark_data_trial188_29692 2026-02-25T22:22:34.309 INFO:tasks.workunit.client.0.trial188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-25T22:22:34.309 INFO:tasks.workunit.client.0.trial188.stdout: 0 0 0 0 0 0 - 0 2026-02-25T22:22:35.309 INFO:tasks.workunit.client.0.trial188.stdout: 1 16 10375 10359 40.4612 40.4648 0.00156649 0.00154252 2026-02-25T22:22:36.309 INFO:tasks.workunit.client.0.trial188.stdout: 2 16 21144 21128 41.262 42.0664 0.00168476 0.00151283 2026-02-25T22:22:37.309 INFO:tasks.workunit.client.0.trial188.stdout: 3 16 31884 31868 41.4913 41.9531 0.00121116 0.00150471 2026-02-25T22:22:38.309 INFO:tasks.workunit.client.0.trial188.stdout: 4 16 42995 42979 41.9682 43.4023 0.00106054 0.00148772 2026-02-25T22:22:39.309 INFO:tasks.workunit.client.0.trial188.stdout: 5 16 54129 54113 42.2722 43.4922 0.00142846 0.00147699 2026-02-25T22:22:40.309 INFO:tasks.workunit.client.0.trial188.stdout: 6 16 64748 64732 42.1397 41.4805 0.00169296 0.00148172 2026-02-25T22:22:41.309 INFO:tasks.workunit.client.0.trial188.stdout: 7 16 75390 75374 42.0579 41.5703 0.000855091 0.00148455 2026-02-25T22:22:42.309 INFO:tasks.workunit.client.0.trial188.stdout: 8 16 85530 85514 41.7514 39.6094 0.00157266 0.00149552 2026-02-25T22:22:43.309 INFO:tasks.workunit.client.0.trial188.stdout: 9 16 95103 95087 41.267 37.3945 0.00166446 0.00151311 2026-02-25T22:22:44.309 INFO:tasks.workunit.client.0.trial188.stdout: 10 7 104219 104212 40.7045 35.6445 0.00117317 0.00153406 2026-02-25T22:22:45.310 INFO:tasks.workunit.client.0.trial188.stdout:Total time run: 10.0012 2026-02-25T22:22:45.310 INFO:tasks.workunit.client.0.trial188.stdout:Total writes made: 104219 2026-02-25T22:22:45.310 INFO:tasks.workunit.client.0.trial188.stdout:Write size: 4096 2026-02-25T22:22:45.310 INFO:tasks.workunit.client.0.trial188.stdout:Object size: 4096 2026-02-25T22:22:45.310 INFO:tasks.workunit.client.0.trial188.stdout:Bandwidth (MB/sec): 40.7058 2026-02-25T22:22:45.310 INFO:tasks.workunit.client.0.trial188.stdout:Stddev Bandwidth: 2.5305 2026-02-25T22:22:45.310 INFO:tasks.workunit.client.0.trial188.stdout:Max bandwidth (MB/sec): 43.4922 2026-02-25T22:22:45.310 INFO:tasks.workunit.client.0.trial188.stdout:Min bandwidth (MB/sec): 35.6445 2026-02-25T22:22:45.311 INFO:tasks.workunit.client.0.trial188.stdout:Average IOPS: 10420 2026-02-25T22:22:45.311 INFO:tasks.workunit.client.0.trial188.stdout:Stddev IOPS: 647.809 2026-02-25T22:22:45.311 INFO:tasks.workunit.client.0.trial188.stdout:Max IOPS: 11134 2026-02-25T22:22:45.311 INFO:tasks.workunit.client.0.trial188.stdout:Min IOPS: 9125 2026-02-25T22:22:45.311 INFO:tasks.workunit.client.0.trial188.stdout:Average Latency(s): 0.00153405 2026-02-25T22:22:45.311 INFO:tasks.workunit.client.0.trial188.stdout:Stddev Latency(s): 0.000755802 2026-02-25T22:22:45.311 INFO:tasks.workunit.client.0.trial188.stdout:Max latency(s): 0.0142635 2026-02-25T22:22:45.311 INFO:tasks.workunit.client.0.trial188.stdout:Min latency(s): 0.000457711 2026-02-25T22:22:45.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:41: TEST_network_ping_test1: sleep 10 2026-02-25T22:22:55.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:42: TEST_network_ping_test1: flush_pg_stats 2026-02-25T22:22:55.318 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:22:55.318 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:22:56.113 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:22:56.113 INFO:tasks.workunit.client.0.trial188.stderr:1 2026-02-25T22:22:56.113 INFO:tasks.workunit.client.0.trial188.stderr:2' 2026-02-25T22:22:56.113 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:22:56.113 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:22:56.117 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:22:56.425 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836490 2026-02-25T22:22:56.426 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836490 2026-02-25T22:22:56.426 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836490' 2026-02-25T22:22:56.426 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:22:56.426 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:22:56.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705673 2026-02-25T22:22:56.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705673 2026-02-25T22:22:56.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836490 1-38654705673' 2026-02-25T22:22:56.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:22:56.512 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:22:56.627 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574856 2026-02-25T22:22:56.627 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574856 2026-02-25T22:22:56.627 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836490 1-38654705673 2-55834574856' 2026-02-25T22:22:56.627 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:22:56.628 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836490 2026-02-25T22:22:56.628 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:22:56.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:22:56.629 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836490 2026-02-25T22:22:56.629 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:22:56.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836490 2026-02-25T22:22:56.631 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836490 2026-02-25T22:22:56.631 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-25T22:22:56.631 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:22:57.068 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836487 -lt 21474836490 2026-02-25T22:22:57.069 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:22:58.069 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:22:58.070 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:22:58.286 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836490 -lt 21474836490 2026-02-25T22:22:58.286 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:22:58.287 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705673 2026-02-25T22:22:58.287 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:22:58.288 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:22:58.289 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705673 2026-02-25T22:22:58.289 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:22:58.290 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705673 2026-02-25T22:22:58.290 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.1 seq 38654705673 2026-02-25T22:22:58.291 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705673' 2026-02-25T22:22:58.291 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:22:58.516 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705673 -lt 38654705673 2026-02-25T22:22:58.516 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:22:58.517 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574856 2026-02-25T22:22:58.517 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:22:58.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:22:58.519 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574856 2026-02-25T22:22:58.519 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:22:58.520 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574856 2026-02-25T22:22:58.520 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.2 seq 55834574856 2026-02-25T22:22:58.521 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574856' 2026-02-25T22:22:58.521 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:22:58.739 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574856 -lt 55834574856 2026-02-25T22:22:58.739 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:22:58.740 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T22:22:58.740 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:58.740 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:58.741 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:58.741 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:58.741 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:22:58.742 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-osd.0.asok 2026-02-25T22:22:58.742 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:22:58.742 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:44: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-osd.0.asok dump_osd_network 2026-02-25T22:22:58.796 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:22:58.797 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 1000, 2026-02-25T22:22:58.797 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [] 2026-02-25T22:22:58.797 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:22:58.804 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:22:58.804 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T22:22:58.813 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:45: TEST_network_ping_test1: test 0 = 0 2026-02-25T22:22:58.814 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:22:58.814 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: jq .threshold 2026-02-25T22:22:58.824 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:46: TEST_network_ping_test1: test 1000 = 1000 2026-02-25T22:22:58.825 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:22:58.826 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T22:22:58.826 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T22:22:58.826 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T22:22:58.826 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:58.826 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:58.826 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:22:58.826 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-mgr.x.asok 2026-02-25T22:22:58.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:22:58.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:48: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-mgr.x.asok dump_osd_network 2026-02-25T22:22:58.875 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:22:58.876 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 1000, 2026-02-25T22:22:58.876 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [] 2026-02-25T22:22:58.876 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:22:58.883 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:22:58.883 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T22:22:58.893 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:49: TEST_network_ping_test1: test 0 = 0 2026-02-25T22:22:58.894 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:22:58.894 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: jq .threshold 2026-02-25T22:22:58.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:50: TEST_network_ping_test1: test 1000 = 1000 2026-02-25T22:22:58.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:22:58.905 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T22:22:58.905 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:22:58.905 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:22:58.906 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:58.906 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:58.906 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:22:58.906 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-osd.0.asok 2026-02-25T22:22:58.907 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:22:58.907 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:52: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-osd.0.asok dump_osd_network 0 2026-02-25T22:22:58.959 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:22:58.960 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 0, 2026-02-25T22:22:58.960 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [ 2026-02-25T22:22:58.960 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:58.960 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:53 2026", 2026-02-25T22:22:58.960 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:58.960 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:22:58.960 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:22:58.960 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:22:58.960 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:58.961 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.157, 2026-02-25T22:22:58.961 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.173, 2026-02-25T22:22:58.961 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.223 2026-02-25T22:22:58.961 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.961 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:58.961 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.187, 2026-02-25T22:22:58.961 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.422, 2026-02-25T22:22:58.961 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.422 2026-02-25T22:22:58.961 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.961 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:58.961 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.187, 2026-02-25T22:22:58.962 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.422, 2026-02-25T22:22:58.962 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.422 2026-02-25T22:22:58.962 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.962 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.194 2026-02-25T22:22:58.962 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.962 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:58.962 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:53 2026", 2026-02-25T22:22:58.962 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:58.962 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:22:58.962 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:22:58.962 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:22:58.962 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:58.963 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.144, 2026-02-25T22:22:58.963 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.170, 2026-02-25T22:22:58.963 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.216 2026-02-25T22:22:58.963 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.963 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:58.963 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.166, 2026-02-25T22:22:58.963 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.381, 2026-02-25T22:22:58.963 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.381 2026-02-25T22:22:58.963 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.963 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:58.963 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.166, 2026-02-25T22:22:58.963 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.381, 2026-02-25T22:22:58.964 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.381 2026-02-25T22:22:58.964 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.964 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.207 2026-02-25T22:22:58.964 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.964 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:58.964 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:55 2026", 2026-02-25T22:22:58.964 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:58.964 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:22:58.964 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:22:58.964 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:22:58.964 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:58.964 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.115, 2026-02-25T22:22:58.964 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.197, 2026-02-25T22:22:58.965 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.197 2026-02-25T22:22:58.965 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.965 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:58.965 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.129, 2026-02-25T22:22:58.965 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.492, 2026-02-25T22:22:58.965 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.492 2026-02-25T22:22:58.965 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.965 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:58.965 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.129, 2026-02-25T22:22:58.965 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.492, 2026-02-25T22:22:58.965 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.492 2026-02-25T22:22:58.965 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.965 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.169 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:55 2026", 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.121, 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.136, 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.139 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:58.966 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.158, 2026-02-25T22:22:58.967 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.191, 2026-02-25T22:22:58.967 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.191 2026-02-25T22:22:58.967 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.967 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:58.967 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.158, 2026-02-25T22:22:58.967 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.191, 2026-02-25T22:22:58.967 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.191 2026-02-25T22:22:58.967 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:58.967 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.154 2026-02-25T22:22:58.967 INFO:tasks.workunit.client.0.trial188.stdout: } 2026-02-25T22:22:58.967 INFO:tasks.workunit.client.0.trial188.stdout: ] 2026-02-25T22:22:58.967 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:22:58.968 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:22:58.968 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T22:22:58.977 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:53: TEST_network_ping_test1: test 4 = 4 2026-02-25T22:22:58.978 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:22:58.978 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: jq .threshold 2026-02-25T22:22:58.988 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:54: TEST_network_ping_test1: test 0 = 0 2026-02-25T22:22:58.989 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:22:58.989 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T22:22:58.989 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T22:22:58.990 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T22:22:58.990 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:22:58.991 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:22:58.991 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:22:58.991 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-mgr.x.asok 2026-02-25T22:22:58.991 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:22:58.991 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:56: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-mgr.x.asok dump_osd_network 0 2026-02-25T22:22:59.038 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:22:59.038 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 0, 2026-02-25T22:22:59.038 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [ 2026-02-25T22:22:59.038 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:59.039 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:54 2026", 2026-02-25T22:22:59.039 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:59.039 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 1, 2026-02-25T22:22:59.039 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 0, 2026-02-25T22:22:59.039 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:22:59.039 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:59.039 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.100, 2026-02-25T22:22:59.039 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.189, 2026-02-25T22:22:59.039 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.272 2026-02-25T22:22:59.039 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.039 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:59.039 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.093, 2026-02-25T22:22:59.040 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.093, 2026-02-25T22:22:59.040 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.093 2026-02-25T22:22:59.040 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.040 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:59.040 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.106, 2026-02-25T22:22:59.040 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 1.354, 2026-02-25T22:22:59.040 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 1.354 2026-02-25T22:22:59.040 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.040 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.142 2026-02-25T22:22:59.040 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.040 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:59.040 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:53 2026", 2026-02-25T22:22:59.040 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:59.041 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:22:59.041 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:22:59.041 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:22:59.041 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:59.041 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.157, 2026-02-25T22:22:59.041 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.173, 2026-02-25T22:22:59.041 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.223 2026-02-25T22:22:59.041 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.041 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:59.041 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.143, 2026-02-25T22:22:59.041 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.104, 2026-02-25T22:22:59.041 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.104 2026-02-25T22:22:59.041 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.042 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:59.042 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.187, 2026-02-25T22:22:59.042 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.422, 2026-02-25T22:22:59.042 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.422 2026-02-25T22:22:59.042 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.042 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.144 2026-02-25T22:22:59.042 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.042 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:59.042 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:53 2026", 2026-02-25T22:22:59.042 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:59.042 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:22:59.042 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:22:59.043 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:22:59.043 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:59.043 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.144, 2026-02-25T22:22:59.043 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.170, 2026-02-25T22:22:59.043 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.216 2026-02-25T22:22:59.043 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.043 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:59.043 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.130, 2026-02-25T22:22:59.043 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.119, 2026-02-25T22:22:59.043 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.119 2026-02-25T22:22:59.043 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.043 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:59.043 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.166, 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.381, 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.381 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.152 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:54 2026", 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 1, 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 0, 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.121, 2026-02-25T22:22:59.044 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.198, 2026-02-25T22:22:59.045 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.186 2026-02-25T22:22:59.045 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.045 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:59.045 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.112, 2026-02-25T22:22:59.045 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.100, 2026-02-25T22:22:59.045 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.100 2026-02-25T22:22:59.045 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.045 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:59.045 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.127, 2026-02-25T22:22:59.045 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 1.452, 2026-02-25T22:22:59.045 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 1.452 2026-02-25T22:22:59.045 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.046 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.179 2026-02-25T22:22:59.046 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.046 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:59.046 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:55 2026", 2026-02-25T22:22:59.046 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:59.046 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:22:59.046 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:22:59.046 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:22:59.046 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:59.046 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.115, 2026-02-25T22:22:59.046 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.197, 2026-02-25T22:22:59.046 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.197 2026-02-25T22:22:59.046 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.047 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:59.047 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.107, 2026-02-25T22:22:59.047 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.102, 2026-02-25T22:22:59.047 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.102 2026-02-25T22:22:59.047 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.047 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:59.047 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.129, 2026-02-25T22:22:59.047 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.492, 2026-02-25T22:22:59.047 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.492 2026-02-25T22:22:59.047 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.047 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.129 2026-02-25T22:22:59.047 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.047 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:59.048 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:53 2026", 2026-02-25T22:22:59.048 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:59.048 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 2, 2026-02-25T22:22:59.048 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:22:59.048 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:22:59.048 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:59.048 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.166, 2026-02-25T22:22:59.048 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.177, 2026-02-25T22:22:59.048 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.195 2026-02-25T22:22:59.048 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.048 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:59.048 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.148, 2026-02-25T22:22:59.048 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.120, 2026-02-25T22:22:59.049 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.120 2026-02-25T22:22:59.049 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.049 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:59.049 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.175, 2026-02-25T22:22:59.049 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.394, 2026-02-25T22:22:59.049 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.394 2026-02-25T22:22:59.049 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.049 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.173 2026-02-25T22:22:59.049 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.049 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:59.049 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:54 2026", 2026-02-25T22:22:59.049 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:59.049 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 1, 2026-02-25T22:22:59.050 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:22:59.050 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:22:59.050 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:59.050 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.144, 2026-02-25T22:22:59.050 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.193, 2026-02-25T22:22:59.050 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.187 2026-02-25T22:22:59.050 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.050 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:59.050 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.136, 2026-02-25T22:22:59.050 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.119, 2026-02-25T22:22:59.050 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.119 2026-02-25T22:22:59.050 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.050 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:59.051 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.152, 2026-02-25T22:22:59.051 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.980, 2026-02-25T22:22:59.051 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.980 2026-02-25T22:22:59.051 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.051 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.157 2026-02-25T22:22:59.051 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.051 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:59.051 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:54 2026", 2026-02-25T22:22:59.051 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:59.051 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 1, 2026-02-25T22:22:59.051 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:22:59.051 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:22:59.051 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:59.052 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.129, 2026-02-25T22:22:59.052 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.183, 2026-02-25T22:22:59.052 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.163 2026-02-25T22:22:59.052 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.052 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:59.052 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.109, 2026-02-25T22:22:59.052 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.109, 2026-02-25T22:22:59.052 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.109 2026-02-25T22:22:59.052 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.052 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:59.052 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.141, 2026-02-25T22:22:59.052 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.932, 2026-02-25T22:22:59.052 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.932 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.168 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:53 2026", 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 2, 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 0, 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.125, 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.161, 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.177 2026-02-25T22:22:59.053 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.054 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:59.054 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.112, 2026-02-25T22:22:59.054 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.100, 2026-02-25T22:22:59.054 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.100 2026-02-25T22:22:59.054 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.054 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:59.054 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.154, 2026-02-25T22:22:59.054 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.224, 2026-02-25T22:22:59.054 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.224 2026-02-25T22:22:59.054 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.054 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.178 2026-02-25T22:22:59.054 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.054 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:53 2026", 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 2, 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.143, 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.165, 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.170 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.134, 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.113, 2026-02-25T22:22:59.055 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.113 2026-02-25T22:22:59.056 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.056 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:59.056 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.156, 2026-02-25T22:22:59.056 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.311, 2026-02-25T22:22:59.056 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.311 2026-02-25T22:22:59.056 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.056 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.203 2026-02-25T22:22:59.056 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.056 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:59.056 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:53 2026", 2026-02-25T22:22:59.056 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:59.056 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 2, 2026-02-25T22:22:59.056 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 0, 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.129, 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.154, 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.159 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.116, 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.100, 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.100 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.149, 2026-02-25T22:22:59.057 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.275, 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.275 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.153 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:22:55 2026", 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.121, 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.136, 2026-02-25T22:22:59.058 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.139 2026-02-25T22:22:59.059 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.059 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:22:59.059 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.095, 2026-02-25T22:22:59.059 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.095, 2026-02-25T22:22:59.059 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.095 2026-02-25T22:22:59.059 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.059 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:22:59.059 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.158, 2026-02-25T22:22:59.059 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.191, 2026-02-25T22:22:59.059 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.191 2026-02-25T22:22:59.059 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:22:59.059 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.111 2026-02-25T22:22:59.059 INFO:tasks.workunit.client.0.trial188.stdout: } 2026-02-25T22:22:59.060 INFO:tasks.workunit.client.0.trial188.stdout: ] 2026-02-25T22:22:59.060 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:22:59.060 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:22:59.061 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T22:22:59.061 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:57: TEST_network_ping_test1: test 12 = 12 2026-02-25T22:22:59.061 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:22:59.061 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: jq .threshold 2026-02-25T22:22:59.068 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:58: TEST_network_ping_test1: test 0 = 0 2026-02-25T22:22:59.068 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:61: TEST_network_ping_test1: sleep 20 2026-02-25T22:23:19.070 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:62: TEST_network_ping_test1: flush_pg_stats 2026-02-25T22:23:19.070 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:23:19.071 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:23:19.289 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:23:19.289 INFO:tasks.workunit.client.0.trial188.stderr:1 2026-02-25T22:23:19.289 INFO:tasks.workunit.client.0.trial188.stderr:2' 2026-02-25T22:23:19.289 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:23:19.289 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:19.290 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:23:19.367 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836496 2026-02-25T22:23:19.367 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836496 2026-02-25T22:23:19.367 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836496' 2026-02-25T22:23:19.368 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:19.368 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:23:19.447 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705679 2026-02-25T22:23:19.447 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705679 2026-02-25T22:23:19.447 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836496 1-38654705679' 2026-02-25T22:23:19.447 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:23:19.447 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:23:19.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574863 2026-02-25T22:23:19.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574863 2026-02-25T22:23:19.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836496 1-38654705679 2-55834574863' 2026-02-25T22:23:19.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:19.527 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836496 2026-02-25T22:23:19.527 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:19.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:23:19.529 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836496 2026-02-25T22:23:19.529 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:19.530 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836496 2026-02-25T22:23:19.530 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836496' 2026-02-25T22:23:19.531 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836496 2026-02-25T22:23:19.531 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:23:19.748 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836494 -lt 21474836496 2026-02-25T22:23:19.748 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:23:20.750 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:23:20.750 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:23:20.963 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836496 -lt 21474836496 2026-02-25T22:23:20.963 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:20.964 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705679 2026-02-25T22:23:20.964 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:20.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:23:20.966 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705679 2026-02-25T22:23:20.966 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:20.968 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705679 2026-02-25T22:23:20.968 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.1 seq 38654705679 2026-02-25T22:23:20.968 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705679' 2026-02-25T22:23:20.968 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:23:21.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705680 -lt 38654705679 2026-02-25T22:23:21.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:23:21.187 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574863 2026-02-25T22:23:21.187 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:23:21.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:23:21.189 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574863 2026-02-25T22:23:21.189 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:23:21.190 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574863 2026-02-25T22:23:21.190 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574863' 2026-02-25T22:23:21.191 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.2 seq 55834574863 2026-02-25T22:23:21.191 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:23:21.406 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574863 -lt 55834574863 2026-02-25T22:23:21.406 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:23:21.407 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T22:23:21.407 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:23:21.407 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:23:21.408 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:23:21.408 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:23:21.408 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:23:21.408 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-osd.0.asok 2026-02-25T22:23:21.408 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:23:21.408 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:63: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-osd.0.asok dump_osd_network 2026-02-25T22:23:21.462 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:23:21.462 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 1000, 2026-02-25T22:23:21.462 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [] 2026-02-25T22:23:21.462 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:23:21.469 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:23:21.469 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T22:23:21.478 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:64: TEST_network_ping_test1: test 0 = 0 2026-02-25T22:23:21.479 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:23:21.479 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: jq .threshold 2026-02-25T22:23:21.489 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:65: TEST_network_ping_test1: test 1000 = 1000 2026-02-25T22:23:21.489 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:23:21.490 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T22:23:21.490 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T22:23:21.490 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T22:23:21.491 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:23:21.491 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:23:21.492 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:23:21.492 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-mgr.x.asok 2026-02-25T22:23:21.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:23:21.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:67: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-mgr.x.asok dump_osd_network 2026-02-25T22:23:21.538 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:23:21.538 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 1000, 2026-02-25T22:23:21.538 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [] 2026-02-25T22:23:21.538 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:23:21.545 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:23:21.545 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T22:23:21.554 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:68: TEST_network_ping_test1: test 0 = 0 2026-02-25T22:23:21.555 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:23:21.555 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: jq .threshold 2026-02-25T22:23:21.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:69: TEST_network_ping_test1: test 1000 = 1000 2026-02-25T22:23:21.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:23:21.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T22:23:21.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:23:21.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:23:21.567 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:23:21.567 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:23:21.567 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:23:21.567 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-osd.0.asok 2026-02-25T22:23:21.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:23:21.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:71: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-osd.0.asok dump_osd_network 0 2026-02-25T22:23:21.619 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:23:21.619 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 0, 2026-02-25T22:23:21.619 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [ 2026-02-25T22:23:21.619 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.619 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:18 2026", 2026-02-25T22:23:21.619 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.619 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:23:21.619 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:23:21.619 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:23:21.619 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.620 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.157, 2026-02-25T22:23:21.620 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.167, 2026-02-25T22:23:21.620 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.196 2026-02-25T22:23:21.620 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.620 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.620 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.180, 2026-02-25T22:23:21.620 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.209, 2026-02-25T22:23:21.620 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.422 2026-02-25T22:23:21.620 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.620 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.620 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.180, 2026-02-25T22:23:21.620 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.209, 2026-02-25T22:23:21.621 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.422 2026-02-25T22:23:21.621 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.621 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.188 2026-02-25T22:23:21.621 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.621 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.621 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:18 2026", 2026-02-25T22:23:21.621 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.621 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:23:21.621 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:23:21.621 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:23:21.621 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.621 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.137, 2026-02-25T22:23:21.621 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.160, 2026-02-25T22:23:21.622 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.190 2026-02-25T22:23:21.622 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.622 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.622 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.148, 2026-02-25T22:23:21.622 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.269, 2026-02-25T22:23:21.622 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.381 2026-02-25T22:23:21.622 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.622 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.622 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.148, 2026-02-25T22:23:21.622 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.269, 2026-02-25T22:23:21.622 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.381 2026-02-25T22:23:21.622 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.622 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.200 2026-02-25T22:23:21.623 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.623 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.623 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:20 2026", 2026-02-25T22:23:21.623 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.623 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:23:21.623 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:23:21.623 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:23:21.623 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.623 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.176, 2026-02-25T22:23:21.623 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.161, 2026-02-25T22:23:21.623 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.185 2026-02-25T22:23:21.623 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.623 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.624 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.235, 2026-02-25T22:23:21.624 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.235, 2026-02-25T22:23:21.624 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.492 2026-02-25T22:23:21.624 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.624 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.624 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.235, 2026-02-25T22:23:21.624 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.235, 2026-02-25T22:23:21.624 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.492 2026-02-25T22:23:21.624 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.624 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.216 2026-02-25T22:23:21.624 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.624 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.624 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:20 2026", 2026-02-25T22:23:21.625 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.625 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:23:21.625 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:23:21.625 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:23:21.625 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.625 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.168, 2026-02-25T22:23:21.625 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.142, 2026-02-25T22:23:21.625 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.140 2026-02-25T22:23:21.625 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.625 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.625 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.281, 2026-02-25T22:23:21.625 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.281, 2026-02-25T22:23:21.625 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.281 2026-02-25T22:23:21.626 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.626 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.626 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.281, 2026-02-25T22:23:21.626 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.281, 2026-02-25T22:23:21.626 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.281 2026-02-25T22:23:21.626 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.626 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.230 2026-02-25T22:23:21.626 INFO:tasks.workunit.client.0.trial188.stdout: } 2026-02-25T22:23:21.626 INFO:tasks.workunit.client.0.trial188.stdout: ] 2026-02-25T22:23:21.626 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:23:21.627 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:23:21.627 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T22:23:21.636 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:72: TEST_network_ping_test1: test 4 = 4 2026-02-25T22:23:21.637 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:23:21.637 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: jq .threshold 2026-02-25T22:23:21.646 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:73: TEST_network_ping_test1: test 0 = 0 2026-02-25T22:23:21.646 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:23:21.647 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T22:23:21.647 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T22:23:21.647 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T22:23:21.648 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:23:21.648 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:23:21.648 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:23:21.648 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-mgr.x.asok 2026-02-25T22:23:21.649 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:23:21.649 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:75: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-mgr.x.asok dump_osd_network 0 2026-02-25T22:23:21.694 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:23:21.695 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 0, 2026-02-25T22:23:21.695 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [ 2026-02-25T22:23:21.695 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.695 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:19 2026", 2026-02-25T22:23:21.695 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.695 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 1, 2026-02-25T22:23:21.695 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 0, 2026-02-25T22:23:21.695 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:23:21.695 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.695 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.139, 2026-02-25T22:23:21.696 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.112, 2026-02-25T22:23:21.696 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.205 2026-02-25T22:23:21.696 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.696 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.696 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.113, 2026-02-25T22:23:21.696 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.079, 2026-02-25T22:23:21.696 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.079 2026-02-25T22:23:21.696 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.696 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.696 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.176, 2026-02-25T22:23:21.696 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.176, 2026-02-25T22:23:21.696 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 1.354 2026-02-25T22:23:21.697 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.697 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.176 2026-02-25T22:23:21.697 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.697 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.697 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:18 2026", 2026-02-25T22:23:21.697 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.697 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:23:21.697 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:23:21.697 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:23:21.697 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.697 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.157, 2026-02-25T22:23:21.697 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.167, 2026-02-25T22:23:21.697 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.196 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.137, 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.130, 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.104 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.180, 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.209, 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.422 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.180 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.698 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.699 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:18 2026", 2026-02-25T22:23:21.699 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.699 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:23:21.699 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:23:21.699 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:23:21.699 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.699 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.137, 2026-02-25T22:23:21.699 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.160, 2026-02-25T22:23:21.699 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.190 2026-02-25T22:23:21.699 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.699 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.699 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.124, 2026-02-25T22:23:21.699 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.120, 2026-02-25T22:23:21.700 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.119 2026-02-25T22:23:21.700 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.700 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.700 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.148, 2026-02-25T22:23:21.700 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.269, 2026-02-25T22:23:21.700 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.381 2026-02-25T22:23:21.700 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.700 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.142 2026-02-25T22:23:21.700 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.700 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.700 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:15 2026", 2026-02-25T22:23:21.700 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.700 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.153, 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.149, 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.186 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.134, 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.107, 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.102 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.701 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.165, 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.222, 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.492 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.162 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:18 2026", 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 2, 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.144, 2026-02-25T22:23:21.702 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.157, 2026-02-25T22:23:21.703 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.179 2026-02-25T22:23:21.703 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.703 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.703 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.128, 2026-02-25T22:23:21.703 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.109, 2026-02-25T22:23:21.703 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.109 2026-02-25T22:23:21.703 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.703 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.703 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.150, 2026-02-25T22:23:21.703 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.202, 2026-02-25T22:23:21.703 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.394 2026-02-25T22:23:21.703 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.703 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.239 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:19 2026", 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 1, 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.154, 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.149, 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.176 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.704 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.140, 2026-02-25T22:23:21.705 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.119, 2026-02-25T22:23:21.705 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.119 2026-02-25T22:23:21.705 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.705 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.705 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.164, 2026-02-25T22:23:21.705 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.192, 2026-02-25T22:23:21.705 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.980 2026-02-25T22:23:21.705 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.705 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.164 2026-02-25T22:23:21.705 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.705 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.705 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:19 2026", 2026-02-25T22:23:21.705 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.706 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 1, 2026-02-25T22:23:21.706 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 0, 2026-02-25T22:23:21.706 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:23:21.706 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.706 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.129, 2026-02-25T22:23:21.706 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.130, 2026-02-25T22:23:21.706 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.169 2026-02-25T22:23:21.706 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.706 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.706 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.100, 2026-02-25T22:23:21.706 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.100, 2026-02-25T22:23:21.706 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.100 2026-02-25T22:23:21.706 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.159, 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.179, 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 1.452 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.131 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:18 2026", 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 2, 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:23:21.707 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.708 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.165, 2026-02-25T22:23:21.708 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.164, 2026-02-25T22:23:21.708 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.167 2026-02-25T22:23:21.708 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.708 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.708 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.145, 2026-02-25T22:23:21.708 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.119, 2026-02-25T22:23:21.708 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.113 2026-02-25T22:23:21.708 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.708 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.708 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.178, 2026-02-25T22:23:21.708 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.240, 2026-02-25T22:23:21.708 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.311 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.225 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:18 2026", 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 2, 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 0, 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.126, 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.136, 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.161 2026-02-25T22:23:21.709 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.118, 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.098, 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.098 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.136, 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.215, 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.224 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.211 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.710 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:19 2026", 2026-02-25T22:23:21.711 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.711 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 1, 2026-02-25T22:23:21.711 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:23:21.711 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:23:21.711 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.711 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.123, 2026-02-25T22:23:21.711 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.125, 2026-02-25T22:23:21.711 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.154 2026-02-25T22:23:21.711 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.711 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.711 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.108, 2026-02-25T22:23:21.711 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.103, 2026-02-25T22:23:21.711 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.103 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.147, 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.168, 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.932 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.145 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:18 2026", 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 2, 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 0, 2026-02-25T22:23:21.712 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:23:21.713 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.713 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.109, 2026-02-25T22:23:21.713 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.120, 2026-02-25T22:23:21.713 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.145 2026-02-25T22:23:21.713 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.713 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.713 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.103, 2026-02-25T22:23:21.713 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.096, 2026-02-25T22:23:21.713 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.096 2026-02-25T22:23:21.713 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.713 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.713 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.118, 2026-02-25T22:23:21.713 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.178, 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.275 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.195 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:23:15 2026", 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.115, 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.133, 2026-02-25T22:23:21.714 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.138 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.106, 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.095, 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.095 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.126, 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.237, 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.237 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.117 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: } 2026-02-25T22:23:21.715 INFO:tasks.workunit.client.0.trial188.stdout: ] 2026-02-25T22:23:21.716 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:23:21.716 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:23:21.717 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T22:23:21.717 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:76: TEST_network_ping_test1: test 12 = 12 2026-02-25T22:23:21.717 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:23:21.717 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: jq .threshold 2026-02-25T22:23:21.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:77: TEST_network_ping_test1: test 0 = 0 2026-02-25T22:23:21.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:81: TEST_network_ping_test1: sleep 50 2026-02-25T22:24:11.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:82: TEST_network_ping_test1: flush_pg_stats 2026-02-25T22:24:11.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:24:11.726 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:24:11.939 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:24:11.939 INFO:tasks.workunit.client.0.trial188.stderr:1 2026-02-25T22:24:11.939 INFO:tasks.workunit.client.0.trial188.stderr:2' 2026-02-25T22:24:11.939 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:24:11.939 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:11.939 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:24:12.018 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836509 2026-02-25T22:24:12.018 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836509 2026-02-25T22:24:12.018 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509' 2026-02-25T22:24:12.018 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:12.019 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:24:12.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705692 2026-02-25T22:24:12.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705692 2026-02-25T22:24:12.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-38654705692' 2026-02-25T22:24:12.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:24:12.098 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:24:12.180 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574875 2026-02-25T22:24:12.180 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574875 2026-02-25T22:24:12.180 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836509 1-38654705692 2-55834574875' 2026-02-25T22:24:12.180 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:12.181 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836509 2026-02-25T22:24:12.181 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:12.181 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:24:12.182 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836509 2026-02-25T22:24:12.182 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:12.184 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836509 2026-02-25T22:24:12.184 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836509' 2026-02-25T22:24:12.184 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836509 2026-02-25T22:24:12.184 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:24:12.396 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836507 -lt 21474836509 2026-02-25T22:24:12.397 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:24:13.398 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:24:13.398 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:24:13.610 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836507 -lt 21474836509 2026-02-25T22:24:13.610 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:24:14.611 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:24:14.612 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:24:14.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836509 -lt 21474836509 2026-02-25T22:24:14.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:14.828 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705692 2026-02-25T22:24:14.828 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:14.829 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:24:14.830 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705692 2026-02-25T22:24:14.830 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:14.831 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705692 2026-02-25T22:24:14.831 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.1 seq 38654705692 2026-02-25T22:24:14.832 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705692' 2026-02-25T22:24:14.832 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:24:15.056 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705692 -lt 38654705692 2026-02-25T22:24:15.056 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:24:15.056 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574875 2026-02-25T22:24:15.057 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:24:15.058 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:24:15.059 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574875 2026-02-25T22:24:15.059 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:24:15.060 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574875 2026-02-25T22:24:15.060 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574875' 2026-02-25T22:24:15.061 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.2 seq 55834574875 2026-02-25T22:24:15.061 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:24:15.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574876 -lt 55834574875 2026-02-25T22:24:15.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:24:15.272 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T22:24:15.272 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:24:15.272 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:24:15.273 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:24:15.273 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:15.273 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:15.274 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-osd.0.asok 2026-02-25T22:24:15.274 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:24:15.274 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:83: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-osd.0.asok dump_osd_network 2026-02-25T22:24:15.329 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:24:15.329 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 1000, 2026-02-25T22:24:15.329 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [] 2026-02-25T22:24:15.329 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:24:15.336 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:24:15.336 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T22:24:15.345 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:84: TEST_network_ping_test1: test 0 = 0 2026-02-25T22:24:15.346 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:24:15.347 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: jq .threshold 2026-02-25T22:24:15.356 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:85: TEST_network_ping_test1: test 1000 = 1000 2026-02-25T22:24:15.357 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:24:15.357 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T22:24:15.357 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T22:24:15.357 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T22:24:15.358 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:24:15.358 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:15.359 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:15.359 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-mgr.x.asok 2026-02-25T22:24:15.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:24:15.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:87: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-mgr.x.asok dump_osd_network 2026-02-25T22:24:15.407 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:24:15.407 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 1000, 2026-02-25T22:24:15.407 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [] 2026-02-25T22:24:15.407 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:24:15.414 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:24:15.414 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T22:24:15.424 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:88: TEST_network_ping_test1: test 0 = 0 2026-02-25T22:24:15.425 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:24:15.425 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: jq .threshold 2026-02-25T22:24:15.435 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:89: TEST_network_ping_test1: test 1000 = 1000 2026-02-25T22:24:15.436 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:24:15.437 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T22:24:15.437 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:24:15.437 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:24:15.438 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:24:15.438 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:15.438 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:15.438 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-osd.0.asok 2026-02-25T22:24:15.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:24:15.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:91: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-osd.0.asok dump_osd_network 0 2026-02-25T22:24:15.491 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:24:15.491 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 0, 2026-02-25T22:24:15.491 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [ 2026-02-25T22:24:15.491 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.491 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:13 2026", 2026-02-25T22:24:15.491 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.491 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:24:15.491 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:24:15.491 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:24:15.492 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.492 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.209, 2026-02-25T22:24:15.492 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.175, 2026-02-25T22:24:15.492 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.170 2026-02-25T22:24:15.492 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.492 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.492 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.289, 2026-02-25T22:24:15.492 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.289, 2026-02-25T22:24:15.492 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.289 2026-02-25T22:24:15.492 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.492 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.492 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.289, 2026-02-25T22:24:15.492 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.289, 2026-02-25T22:24:15.493 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.289 2026-02-25T22:24:15.493 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.493 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.260 2026-02-25T22:24:15.493 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.493 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.493 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:13 2026", 2026-02-25T22:24:15.493 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.493 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:24:15.493 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:24:15.493 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:24:15.493 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.493 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.209, 2026-02-25T22:24:15.494 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.175, 2026-02-25T22:24:15.494 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.169 2026-02-25T22:24:15.494 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.494 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.494 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.242, 2026-02-25T22:24:15.494 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.242, 2026-02-25T22:24:15.494 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.307 2026-02-25T22:24:15.494 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.494 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.494 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.242, 2026-02-25T22:24:15.494 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.242, 2026-02-25T22:24:15.494 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.307 2026-02-25T22:24:15.494 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.495 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.201 2026-02-25T22:24:15.495 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.495 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.495 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:10 2026", 2026-02-25T22:24:15.495 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.495 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:24:15.495 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:24:15.495 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:24:15.495 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.495 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.151, 2026-02-25T22:24:15.495 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.153, 2026-02-25T22:24:15.495 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.157 2026-02-25T22:24:15.495 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.496 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.496 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.166, 2026-02-25T22:24:15.496 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.183, 2026-02-25T22:24:15.496 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.326 2026-02-25T22:24:15.496 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.496 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.496 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.166, 2026-02-25T22:24:15.496 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.183, 2026-02-25T22:24:15.496 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.326 2026-02-25T22:24:15.496 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.496 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.242 2026-02-25T22:24:15.496 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.496 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.497 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:10 2026", 2026-02-25T22:24:15.497 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.497 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:24:15.497 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:24:15.497 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:24:15.497 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.497 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.151, 2026-02-25T22:24:15.497 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.157, 2026-02-25T22:24:15.497 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.151 2026-02-25T22:24:15.497 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.497 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.497 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.199, 2026-02-25T22:24:15.497 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.200, 2026-02-25T22:24:15.498 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.341 2026-02-25T22:24:15.498 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.498 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.498 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.199, 2026-02-25T22:24:15.498 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.200, 2026-02-25T22:24:15.498 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.341 2026-02-25T22:24:15.498 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.498 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.224 2026-02-25T22:24:15.498 INFO:tasks.workunit.client.0.trial188.stdout: } 2026-02-25T22:24:15.498 INFO:tasks.workunit.client.0.trial188.stdout: ] 2026-02-25T22:24:15.498 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:24:15.499 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:24:15.499 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T22:24:15.508 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:92: TEST_network_ping_test1: test 4 = 4 2026-02-25T22:24:15.509 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:24:15.510 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: jq .threshold 2026-02-25T22:24:15.520 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:93: TEST_network_ping_test1: test 0 = 0 2026-02-25T22:24:15.520 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:24:15.521 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T22:24:15.521 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T22:24:15.521 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T22:24:15.522 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:24:15.522 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:15.522 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:15.522 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-mgr.x.asok 2026-02-25T22:24:15.522 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:24:15.522 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:95: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-mgr.x.asok dump_osd_network 0 2026-02-25T22:24:15.570 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:24:15.570 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 0, 2026-02-25T22:24:15.570 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [ 2026-02-25T22:24:15.570 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.570 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:08 2026", 2026-02-25T22:24:15.571 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.571 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:24:15.571 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:24:15.571 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:24:15.571 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.571 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.179, 2026-02-25T22:24:15.571 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.167, 2026-02-25T22:24:15.571 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.168 2026-02-25T22:24:15.571 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.571 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.571 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.173, 2026-02-25T22:24:15.571 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.127, 2026-02-25T22:24:15.572 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.120 2026-02-25T22:24:15.572 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.572 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.572 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.183, 2026-02-25T22:24:15.572 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.203, 2026-02-25T22:24:15.572 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.286 2026-02-25T22:24:15.572 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.572 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.205 2026-02-25T22:24:15.572 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.572 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.572 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:13 2026", 2026-02-25T22:24:15.572 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.572 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 2, 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.168, 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.154, 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.153 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.122, 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.116, 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.112 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.573 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.230, 2026-02-25T22:24:15.574 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.230, 2026-02-25T22:24:15.574 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.239 2026-02-25T22:24:15.574 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.574 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.130 2026-02-25T22:24:15.574 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.574 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.574 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:13 2026", 2026-02-25T22:24:15.574 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.574 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 2, 2026-02-25T22:24:15.574 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 0, 2026-02-25T22:24:15.574 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:24:15.574 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.574 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.166, 2026-02-25T22:24:15.575 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.140, 2026-02-25T22:24:15.575 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.139 2026-02-25T22:24:15.575 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.575 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.575 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.149, 2026-02-25T22:24:15.575 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.110, 2026-02-25T22:24:15.575 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.110 2026-02-25T22:24:15.575 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.575 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.575 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.187, 2026-02-25T22:24:15.575 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.187, 2026-02-25T22:24:15.575 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.215 2026-02-25T22:24:15.575 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.576 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.162 2026-02-25T22:24:15.576 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.576 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.576 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:08 2026", 2026-02-25T22:24:15.576 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.576 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:24:15.576 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:24:15.576 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:24:15.576 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.576 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.158, 2026-02-25T22:24:15.576 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.164, 2026-02-25T22:24:15.576 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.166 2026-02-25T22:24:15.576 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.141, 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.138, 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.120 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.168, 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.191, 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.307 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.216 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.577 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:10 2026", 2026-02-25T22:24:15.578 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.578 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:24:15.578 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:24:15.578 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:24:15.578 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.578 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.151, 2026-02-25T22:24:15.578 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.153, 2026-02-25T22:24:15.578 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.157 2026-02-25T22:24:15.578 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.578 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.578 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.136, 2026-02-25T22:24:15.578 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.136, 2026-02-25T22:24:15.578 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.116 2026-02-25T22:24:15.579 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.579 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.579 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.166, 2026-02-25T22:24:15.579 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.183, 2026-02-25T22:24:15.579 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.326 2026-02-25T22:24:15.579 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.579 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.244 2026-02-25T22:24:15.579 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.579 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.579 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:10 2026", 2026-02-25T22:24:15.579 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.579 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 0, 2026-02-25T22:24:15.579 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:24:15.580 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:24:15.580 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.580 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.151, 2026-02-25T22:24:15.580 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.157, 2026-02-25T22:24:15.580 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.151 2026-02-25T22:24:15.580 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.580 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.580 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.126, 2026-02-25T22:24:15.580 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.125, 2026-02-25T22:24:15.580 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.103 2026-02-25T22:24:15.580 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.580 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.580 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.199, 2026-02-25T22:24:15.581 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.200, 2026-02-25T22:24:15.581 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.341 2026-02-25T22:24:15.581 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.581 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.231 2026-02-25T22:24:15.581 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.581 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.581 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:13 2026", 2026-02-25T22:24:15.581 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.581 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 2, 2026-02-25T22:24:15.581 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 1, 2026-02-25T22:24:15.581 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:24:15.581 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.581 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.142, 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.146, 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.155 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.121, 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.120, 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.119 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.161, 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.171, 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.240 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.582 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.148 2026-02-25T22:24:15.583 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.583 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.583 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:13 2026", 2026-02-25T22:24:15.583 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.583 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 2, 2026-02-25T22:24:15.583 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 0, 2026-02-25T22:24:15.583 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:24:15.583 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.583 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.144, 2026-02-25T22:24:15.583 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.122, 2026-02-25T22:24:15.583 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.116 2026-02-25T22:24:15.583 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.583 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.107, 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.097, 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.085 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.196, 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.196, 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.196 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.109 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:09 2026", 2026-02-25T22:24:15.584 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.585 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 1, 2026-02-25T22:24:15.585 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:24:15.585 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:24:15.585 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.585 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.102, 2026-02-25T22:24:15.585 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.115, 2026-02-25T22:24:15.585 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.130 2026-02-25T22:24:15.585 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.585 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.585 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.098, 2026-02-25T22:24:15.585 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.098, 2026-02-25T22:24:15.585 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.098 2026-02-25T22:24:15.585 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.109, 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.142, 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.192 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.116 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:09 2026", 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 1, 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 0, 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "front", 2026-02-25T22:24:15.586 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.587 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.121, 2026-02-25T22:24:15.587 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.126, 2026-02-25T22:24:15.587 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.128 2026-02-25T22:24:15.587 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.587 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.587 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.115, 2026-02-25T22:24:15.587 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.103, 2026-02-25T22:24:15.587 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.095 2026-02-25T22:24:15.587 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.587 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.587 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.132, 2026-02-25T22:24:15.587 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.151, 2026-02-25T22:24:15.587 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.179 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.152 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:09 2026", 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 1, 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 2, 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.124, 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.111, 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.120 2026-02-25T22:24:15.588 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.113, 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.094, 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.094 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.131, 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.138, 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.168 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.108 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: { 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: "last update": "Wed Feb 25 22:24:09 2026", 2026-02-25T22:24:15.589 INFO:tasks.workunit.client.0.trial188.stdout: "stale": false, 2026-02-25T22:24:15.590 INFO:tasks.workunit.client.0.trial188.stdout: "from osd": 1, 2026-02-25T22:24:15.590 INFO:tasks.workunit.client.0.trial188.stdout: "to osd": 0, 2026-02-25T22:24:15.590 INFO:tasks.workunit.client.0.trial188.stdout: "interface": "back", 2026-02-25T22:24:15.590 INFO:tasks.workunit.client.0.trial188.stdout: "average": { 2026-02-25T22:24:15.590 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.093, 2026-02-25T22:24:15.590 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.104, 2026-02-25T22:24:15.590 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.106 2026-02-25T22:24:15.590 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.590 INFO:tasks.workunit.client.0.trial188.stdout: "min": { 2026-02-25T22:24:15.590 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.092, 2026-02-25T22:24:15.590 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.078, 2026-02-25T22:24:15.590 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.077 2026-02-25T22:24:15.590 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.591 INFO:tasks.workunit.client.0.trial188.stdout: "max": { 2026-02-25T22:24:15.591 INFO:tasks.workunit.client.0.trial188.stdout: "1min": 0.099, 2026-02-25T22:24:15.591 INFO:tasks.workunit.client.0.trial188.stdout: "5min": 0.133, 2026-02-25T22:24:15.591 INFO:tasks.workunit.client.0.trial188.stdout: "15min": 0.176 2026-02-25T22:24:15.591 INFO:tasks.workunit.client.0.trial188.stdout: }, 2026-02-25T22:24:15.591 INFO:tasks.workunit.client.0.trial188.stdout: "last": 0.094 2026-02-25T22:24:15.591 INFO:tasks.workunit.client.0.trial188.stdout: } 2026-02-25T22:24:15.591 INFO:tasks.workunit.client.0.trial188.stdout: ] 2026-02-25T22:24:15.591 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:24:15.592 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:24:15.592 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: jq '.entries | length' 2026-02-25T22:24:15.592 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:96: TEST_network_ping_test1: test 12 = 12 2026-02-25T22:24:15.592 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:24:15.592 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: jq .threshold 2026-02-25T22:24:15.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:97: TEST_network_ping_test1: test 0 = 0 2026-02-25T22:24:15.600 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:24:15.601 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: get_asok_path mgr.x 2026-02-25T22:24:15.601 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mgr.x 2026-02-25T22:24:15.601 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mgr.x ']' 2026-02-25T22:24:15.601 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:24:15.601 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:15.601 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:15.602 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-mgr.x.asok 2026-02-25T22:24:15.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:24:15.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:100: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-mgr.x.asok dump_osd_network 99 2026-02-25T22:24:15.650 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:24:15.650 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 99, 2026-02-25T22:24:15.650 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [] 2026-02-25T22:24:15.650 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:24:15.657 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:24:15.657 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: jq .threshold 2026-02-25T22:24:15.668 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:101: TEST_network_ping_test1: test 99 = 99 2026-02-25T22:24:15.668 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: tee td/network-ping/json 2026-02-25T22:24:15.669 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: get_asok_path osd.0 2026-02-25T22:24:15.669 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:24:15.669 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:24:15.669 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:24:15.669 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:15.669 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:15.670 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-osd.0.asok 2026-02-25T22:24:15.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: CEPH_ARGS= 2026-02-25T22:24:15.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:102: TEST_network_ping_test1: ceph daemon /tmp/ceph-asok.24105/ceph-osd.0.asok dump_osd_network 98 2026-02-25T22:24:15.725 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:24:15.725 INFO:tasks.workunit.client.0.trial188.stdout: "threshold": 98, 2026-02-25T22:24:15.725 INFO:tasks.workunit.client.0.trial188.stdout: "entries": [] 2026-02-25T22:24:15.725 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:24:15.732 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: cat td/network-ping/json 2026-02-25T22:24:15.732 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: jq .threshold 2026-02-25T22:24:15.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:103: TEST_network_ping_test1: test 98 = 98 2026-02-25T22:24:15.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:105: TEST_network_ping_test1: rm -f td/network-ping/json 2026-02-25T22:24:15.742 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2026-02-25T22:24:15.742 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-25T22:24:15.742 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:24:15.743 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-25T22:24:15.743 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:24:15.743 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:24:15.744 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:24:15.744 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:24:15.744 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:24:16.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:24:16.054 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:24:16.055 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:24:16.056 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:24:16.057 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:24:16.058 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:24:16.058 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:24:16.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:24:16.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:24:16.060 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:24:16.060 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:24:16.061 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:24:16.062 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:24:16.062 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-25T22:24:16.639 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:24:16.639 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:16.640 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:16.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.24105 2026-02-25T22:24:16.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:24:16.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:24:16.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:17: run: for func in $funcs 2026-02-25T22:24:16.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:18: run: setup td/network-ping 2026-02-25T22:24:16.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/network-ping 2026-02-25T22:24:16.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/network-ping 2026-02-25T22:24:16.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-25T22:24:16.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:24:16.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-25T22:24:16.643 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:24:16.643 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:24:16.643 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:24:16.643 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:24:16.643 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:24:16.644 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:24:16.645 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:24:16.645 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:24:16.645 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:24:16.646 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:24:16.646 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:24:16.647 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:24:16.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:24:16.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:24:16.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:24:16.649 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:24:16.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:24:16.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:24:16.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-25T22:24:16.652 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:24:16.652 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:16.652 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:16.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.24105 2026-02-25T22:24:16.653 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:24:16.653 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:24:16.653 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/network-ping 2026-02-25T22:24:16.654 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:24:16.655 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:16.655 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:16.655 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.24105 2026-02-25T22:24:16.656 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:24:16.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:24:16.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:24:16.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/network-ping 1' TERM HUP INT 2026-02-25T22:24:16.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:19: run: TEST_network_ping_test2 td/network-ping 2026-02-25T22:24:16.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:111: TEST_network_ping_test2: local dir=td/network-ping 2026-02-25T22:24:16.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:113: TEST_network_ping_test2: export CEPH_ARGS 2026-02-25T22:24:16.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: export 'EXTRA_OPTS= --mon_warn_on_slow_ping_time=0.001' 2026-02-25T22:24:16.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:114: TEST_network_ping_test2: EXTRA_OPTS=' --mon_warn_on_slow_ping_time=0.001' 2026-02-25T22:24:16.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:115: TEST_network_ping_test2: run_mon td/network-ping a 2026-02-25T22:24:16.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/network-ping 2026-02-25T22:24:16.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:24:16.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:24:16.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:24:16.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/network-ping/a 2026-02-25T22:24:16.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/network-ping/a --run-dir=td/network-ping 2026-02-25T22:24:16.679 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:24:16.679 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:24:16.680 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:24:16.680 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:24:16.681 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:16.681 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:16.681 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:24:16.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --mon_warn_on_slow_ping_time=0.001 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/network-ping/a '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --mon-cluster-log-file=td/network-ping/log --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:24:16.708 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:24:16.709 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:24:16.709 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:24:16.709 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:24:16.709 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:24:16.710 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:24:16.710 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:24:16.710 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:24:16.710 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:24:16.710 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:24:16.711 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:16.711 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:16.711 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-mon.a.asok 2026-02-25T22:24:16.711 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:24:16.711 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.24105/ceph-mon.a.asok config get fsid 2026-02-25T22:24:16.766 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:24:16.766 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:24:16.766 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:24:16.766 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:24:16.767 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:24:16.767 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:24:16.768 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:24:16.768 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:24:16.775 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:24:16.775 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:16.775 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:16.775 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.24105/ceph-mon.a.asok 2026-02-25T22:24:16.775 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:24:16.776 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.24105/ceph-mon.a.asok config get mon_host 2026-02-25T22:24:16.823 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:116: TEST_network_ping_test2: run_mgr td/network-ping x 2026-02-25T22:24:16.823 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/network-ping 2026-02-25T22:24:16.824 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:24:16.824 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:24:16.824 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:24:16.824 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/network-ping/x 2026-02-25T22:24:16.824 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:24:16.939 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:24:16.939 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:24:16.939 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:24:16.939 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:24:16.939 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:16.939 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:16.940 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:24:16.940 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:24:16.941 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --mon_warn_on_slow_ping_time=0.001 --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/network-ping/x '--log-file=td/network-ping/$name.log' '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --run-dir=td/network-ping '--pid-file=td/network-ping/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:24:16.963 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:117: TEST_network_ping_test2: run_osd td/network-ping 0 2026-02-25T22:24:16.963 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-25T22:24:16.963 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:24:16.963 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:24:16.963 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:24:16.963 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/0 2026-02-25T22:24:16.964 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-25T22:24:16.964 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:24:16.964 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:24:16.964 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:24:16.964 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/0' 2026-02-25T22:24:16.964 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/0/journal' 2026-02-25T22:24:16.964 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:24:16.964 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-25T22:24:16.964 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-25T22:24:16.964 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:24:16.964 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:24:16.964 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:24:16.964 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:24:16.965 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:16.965 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:16.965 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:24:16.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:24:16.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:24:16.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:24:16.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:24:16.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-25T22:24:16.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-25T22:24:16.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:24:16.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:24:16.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:24:16.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:24:16.966 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:24:16.966 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:24:16.966 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/0 2026-02-25T22:24:16.966 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:24:16.966 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=96b41b7a-3617-4965-a0b1-22ca97a1cfcb 2026-02-25T22:24:16.967 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 96b41b7a-3617-4965-a0b1-22ca97a1cfcb 2026-02-25T22:24:16.967 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 96b41b7a-3617-4965-a0b1-22ca97a1cfcb' 2026-02-25T22:24:16.967 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:24:16.976 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCQdp9pEO8AOhAAr/82LKTQsGceiXnGXdOtbA== 2026-02-25T22:24:16.977 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCQdp9pEO8AOhAAr/82LKTQsGceiXnGXdOtbA=="}' 2026-02-25T22:24:16.977 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 96b41b7a-3617-4965-a0b1-22ca97a1cfcb -i td/network-ping/0/new.json 2026-02-25T22:24:17.086 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:24:17.094 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/0/new.json 2026-02-25T22:24:17.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCQdp9pEO8AOhAAr/82LKTQsGceiXnGXdOtbA== --osd-uuid 96b41b7a-3617-4965-a0b1-22ca97a1cfcb 2026-02-25T22:24:17.111 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:17.101+0000 7fcc9445fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:17.113 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:17.105+0000 7fcc9445fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:17.113 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:17.105+0000 7fcc9445fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:17.114 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:17.105+0000 7fcc9445fa40 -1 bdev(0x55d21b0da400 td/network-ping/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:24:17.114 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:17.105+0000 7fcc9445fa40 -1 bluestore(td/network-ping/0) _read_fsid unparsable uuid 2026-02-25T22:24:17.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/0/keyring 2026-02-25T22:24:17.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:24:17.841 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:24:17.841 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:24:17.841 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:24:17.967 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:24:17.967 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:24:17.968 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/0 --osd-journal=td/network-ping/0/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:24:17.968 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:24:17.968 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:24:17.968 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:24:17.991 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:17.981+0000 7f7acb197a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:17.994 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:17.985+0000 7f7acb197a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:17.995 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:17.985+0000 7f7acb197a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:18.079 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:24:18.080 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:24:18.080 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:24:18.080 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:24:18.080 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:24:18.080 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:24:18.080 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:24:18.081 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:24:18.081 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:24:18.081 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:24:18.190 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:24:18.518 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:18.509+0000 7f7acb197a40 -1 Falling back to public interface 2026-02-25T22:24:19.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:24:19.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:24:19.192 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:24:19.193 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:24:19.193 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:24:19.193 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:24:19.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:24:19.349 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:19.341+0000 7f7acb197a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:24:20.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:24:20.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:24:20.305 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:24:20.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:24:20.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:24:20.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:24:20.523 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4169651351,v1:127.0.0.1:6801/4169651351] [v2:127.0.0.1:6802/4169651351,v1:127.0.0.1:6803/4169651351] exists,up 96b41b7a-3617-4965-a0b1-22ca97a1cfcb 2026-02-25T22:24:20.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:24:20.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:24:20.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:24:20.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:118: TEST_network_ping_test2: run_osd td/network-ping 1 2026-02-25T22:24:20.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-25T22:24:20.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:24:20.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:24:20.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:24:20.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/1 2026-02-25T22:24:20.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-25T22:24:20.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:24:20.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:24:20.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:24:20.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/1' 2026-02-25T22:24:20.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/1/journal' 2026-02-25T22:24:20.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:24:20.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-25T22:24:20.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-25T22:24:20.527 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:24:20.527 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:24:20.528 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:24:20.528 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:24:20.528 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:20.528 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:20.528 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:24:20.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:24:20.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:24:20.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:24:20.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:24:20.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-25T22:24:20.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-25T22:24:20.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:24:20.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:24:20.529 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:24:20.529 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:24:20.529 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:24:20.529 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:24:20.529 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/1 2026-02-25T22:24:20.529 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:24:20.530 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=446707ae-c9ae-4195-b8c7-a8477ce19e3b 2026-02-25T22:24:20.530 INFO:tasks.workunit.client.0.trial188.stdout:add osd1 446707ae-c9ae-4195-b8c7-a8477ce19e3b 2026-02-25T22:24:20.530 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 446707ae-c9ae-4195-b8c7-a8477ce19e3b' 2026-02-25T22:24:20.530 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:24:20.540 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCUdp9pqlwBIBAA9cR7cq1HYWr5LjkwwnOj1g== 2026-02-25T22:24:20.541 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCUdp9pqlwBIBAA9cR7cq1HYWr5LjkwwnOj1g=="}' 2026-02-25T22:24:20.541 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 446707ae-c9ae-4195-b8c7-a8477ce19e3b -i td/network-ping/1/new.json 2026-02-25T22:24:20.756 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:24:20.766 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/1/new.json 2026-02-25T22:24:20.767 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCUdp9pqlwBIBAA9cR7cq1HYWr5LjkwwnOj1g== --osd-uuid 446707ae-c9ae-4195-b8c7-a8477ce19e3b 2026-02-25T22:24:20.784 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:20.774+0000 7f0f7938ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:20.785 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:20.778+0000 7f0f7938ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:20.786 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:20.778+0000 7f0f7938ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:20.787 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:20.778+0000 7f0f7938ea40 -1 bdev(0x55b02370a400 td/network-ping/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:24:20.787 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:20.778+0000 7f0f7938ea40 -1 bluestore(td/network-ping/1) _read_fsid unparsable uuid 2026-02-25T22:24:21.383 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/1/keyring 2026-02-25T22:24:21.384 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:24:21.385 INFO:tasks.workunit.client.0.trial188.stdout:adding osd1 key to auth repository 2026-02-25T22:24:21.385 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:24:21.385 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:24:21.679 INFO:tasks.workunit.client.0.trial188.stdout:start osd.1 2026-02-25T22:24:21.679 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:24:21.680 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/1 --osd-journal=td/network-ping/1/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:24:21.680 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:24:21.680 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:24:21.680 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:24:21.695 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:21.686+0000 7fd336af0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:21.696 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:21.690+0000 7fd336af0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:21.697 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:21.690+0000 7fd336af0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:21.903 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:24:21.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:24:21.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:24:21.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:24:21.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:24:21.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:24:21.904 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:24:21.905 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:24:21.905 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:24:21.905 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:24:22.082 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:22.074+0000 7fd336af0a40 -1 Falling back to public interface 2026-02-25T22:24:22.115 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:24:22.785 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:22.778+0000 7fd336af0a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:24:23.117 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:24:23.117 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:24:23.117 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:24:23.118 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:24:23.118 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:24:23.118 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:24:23.335 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:24:24.337 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:24:24.337 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:24:24.337 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:24:24.338 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:24:24.338 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:24:24.338 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:24:24.556 INFO:tasks.workunit.client.0.trial188.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2726742542,v1:127.0.0.1:6811/2726742542] [v2:127.0.0.1:6812/2726742542,v1:127.0.0.1:6813/2726742542] exists,up 446707ae-c9ae-4195-b8c7-a8477ce19e3b 2026-02-25T22:24:24.559 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:24:24.559 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:24:24.559 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:24:24.559 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:119: TEST_network_ping_test2: run_osd td/network-ping 2 2026-02-25T22:24:24.559 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/network-ping 2026-02-25T22:24:24.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:24:24.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:24:24.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:24:24.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/network-ping/2 2026-02-25T22:24:24.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 ' 2026-02-25T22:24:24.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:24:24.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:24:24.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:24:24.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/network-ping/2' 2026-02-25T22:24:24.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/network-ping/2/journal' 2026-02-25T22:24:24.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:24:24.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+=' --mon_warn_on_slow_ping_time=0.001' 2026-02-25T22:24:24.561 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/network-ping' 2026-02-25T22:24:24.561 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:24:24.561 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:24:24.561 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:24:24.561 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:24:24.561 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:24:24.561 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:24:24.561 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:24:24.561 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' 2026-02-25T22:24:24.561 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:24:24.561 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:24:24.561 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:24:24.561 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/network-ping/$name.log' 2026-02-25T22:24:24.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/network-ping/$name.pid' 2026-02-25T22:24:24.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:24:24.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:24:24.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:24:24.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:24:24.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:24:24.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:24:24.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/network-ping/2 2026-02-25T22:24:24.563 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:24:24.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=17f5c337-df17-4382-8d6f-44f9c22434f6 2026-02-25T22:24:24.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 17f5c337-df17-4382-8d6f-44f9c22434f6' 2026-02-25T22:24:24.563 INFO:tasks.workunit.client.0.trial188.stdout:add osd2 17f5c337-df17-4382-8d6f-44f9c22434f6 2026-02-25T22:24:24.564 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:24:24.573 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCYdp9pkQz8IRAAb23Y90F3EjTwSw5eWTKdLw== 2026-02-25T22:24:24.574 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCYdp9pkQz8IRAAb23Y90F3EjTwSw5eWTKdLw=="}' 2026-02-25T22:24:24.574 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 17f5c337-df17-4382-8d6f-44f9c22434f6 -i td/network-ping/2/new.json 2026-02-25T22:24:24.792 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:24:24.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/network-ping/2/new.json 2026-02-25T22:24:24.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCYdp9pkQz8IRAAb23Y90F3EjTwSw5eWTKdLw== --osd-uuid 17f5c337-df17-4382-8d6f-44f9c22434f6 2026-02-25T22:24:24.822 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:24.814+0000 7f3759f24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:24.824 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:24.814+0000 7f3759f24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:24.825 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:24.818+0000 7f3759f24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:24.826 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:24.818+0000 7f3759f24a40 -1 bdev(0x559cd999a400 td/network-ping/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:24:24.826 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:24.818+0000 7f3759f24a40 -1 bluestore(td/network-ping/2) _read_fsid unparsable uuid 2026-02-25T22:24:25.451 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/network-ping/2/keyring 2026-02-25T22:24:25.452 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:24:25.452 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:24:25.453 INFO:tasks.workunit.client.0.trial188.stdout:adding osd2 key to auth repository 2026-02-25T22:24:25.453 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/network-ping/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:24:25.750 INFO:tasks.workunit.client.0.trial188.stdout:start osd.2 2026-02-25T22:24:25.750 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:24:25.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=d02688a5-5463-4883-b83c-ce44500bc71b --auth-supported=none --mon-host=127.0.0.1:7146 --debug_disable_randomized_ping=true --debug_heartbeat_testing_span=5 --osd_heartbeat_interval=1 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/network-ping/2 --osd-journal=td/network-ping/2/journal --chdir= --mon_warn_on_slow_ping_time=0.001 --run-dir=td/network-ping '--admin-socket=/tmp/ceph-asok.24105/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/network-ping/$name.log' '--pid-file=td/network-ping/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:24:25.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:24:25.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:24:25.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:24:25.774 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:25.766+0000 7f6017fffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:25.776 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:25.766+0000 7f6017fffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:25.777 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:25.770+0000 7f6017fffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:24:25.959 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:24:25.959 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:24:25.959 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:24:25.959 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:24:25.960 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:24:25.960 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:24:25.960 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:24:25.960 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:24:25.961 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:24:25.961 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:24:26.176 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:24:26.193 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:26.186+0000 7f6017fffa40 -1 Falling back to public interface 2026-02-25T22:24:26.791 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:24:26.782+0000 7f6017fffa40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:24:27.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:24:27.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:24:27.177 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:24:27.178 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:24:27.178 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:24:27.178 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:24:27.395 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:24:28.396 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:24:28.396 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:24:28.397 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:24:28.397 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:24:28.397 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:24:28.397 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:24:28.613 INFO:tasks.workunit.client.0.trial188.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/4116497976,v1:127.0.0.1:6819/4116497976] [v2:127.0.0.1:6820/4116497976,v1:127.0.0.1:6821/4116497976] exists,up 17f5c337-df17-4382-8d6f-44f9c22434f6 2026-02-25T22:24:28.613 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:24:28.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:24:28.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:24:28.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:121: TEST_network_ping_test2: sleep 5 2026-02-25T22:24:33.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:122: TEST_network_ping_test2: ceph osd crush add-bucket dc1 datacenter 2026-02-25T22:24:33.876 INFO:tasks.workunit.client.0.trial188.stderr:bucket 'dc1' already exists 2026-02-25T22:24:33.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:123: TEST_network_ping_test2: ceph osd crush add-bucket dc2 datacenter 2026-02-25T22:24:34.180 INFO:tasks.workunit.client.0.trial188.stderr:bucket 'dc2' already exists 2026-02-25T22:24:34.191 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:124: TEST_network_ping_test2: ceph osd crush add-bucket dc3 datacenter 2026-02-25T22:24:34.453 INFO:tasks.workunit.client.0.trial188.stderr:bucket 'dc3' already exists 2026-02-25T22:24:34.462 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:125: TEST_network_ping_test2: ceph osd crush add-bucket rack1 rack 2026-02-25T22:24:34.728 INFO:tasks.workunit.client.0.trial188.stderr:bucket 'rack1' already exists 2026-02-25T22:24:34.737 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:126: TEST_network_ping_test2: ceph osd crush add-bucket rack2 rack 2026-02-25T22:24:35.007 INFO:tasks.workunit.client.0.trial188.stderr:bucket 'rack2' already exists 2026-02-25T22:24:35.018 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:127: TEST_network_ping_test2: ceph osd crush add-bucket rack3 rack 2026-02-25T22:24:35.282 INFO:tasks.workunit.client.0.trial188.stderr:bucket 'rack3' already exists 2026-02-25T22:24:35.291 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:128: TEST_network_ping_test2: ceph osd crush add-bucket host1 host 2026-02-25T22:24:35.564 INFO:tasks.workunit.client.0.trial188.stderr:bucket 'host1' already exists 2026-02-25T22:24:35.574 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:129: TEST_network_ping_test2: ceph osd crush add-bucket host2 host 2026-02-25T22:24:35.849 INFO:tasks.workunit.client.0.trial188.stderr:bucket 'host2' already exists 2026-02-25T22:24:35.859 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:130: TEST_network_ping_test2: ceph osd crush add-bucket host3 host 2026-02-25T22:24:36.151 INFO:tasks.workunit.client.0.trial188.stderr:bucket 'host3' already exists 2026-02-25T22:24:36.162 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:131: TEST_network_ping_test2: ceph osd crush move dc1 root=default 2026-02-25T22:24:36.429 INFO:tasks.workunit.client.0.trial188.stderr:no need to move item id -5 name 'dc1' to location {root=default} in crush map 2026-02-25T22:24:36.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:132: TEST_network_ping_test2: ceph osd crush move dc2 root=default 2026-02-25T22:24:36.692 INFO:tasks.workunit.client.0.trial188.stderr:no need to move item id -6 name 'dc2' to location {root=default} in crush map 2026-02-25T22:24:36.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:133: TEST_network_ping_test2: ceph osd crush move dc3 root=default 2026-02-25T22:24:36.966 INFO:tasks.workunit.client.0.trial188.stderr:no need to move item id -7 name 'dc3' to location {root=default} in crush map 2026-02-25T22:24:36.976 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:134: TEST_network_ping_test2: ceph osd crush move rack1 datacenter=dc1 2026-02-25T22:24:37.247 INFO:tasks.workunit.client.0.trial188.stderr:no need to move item id -8 name 'rack1' to location {datacenter=dc1} in crush map 2026-02-25T22:24:37.258 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:135: TEST_network_ping_test2: ceph osd crush move rack2 datacenter=dc2 2026-02-25T22:24:37.525 INFO:tasks.workunit.client.0.trial188.stderr:no need to move item id -9 name 'rack2' to location {datacenter=dc2} in crush map 2026-02-25T22:24:37.535 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:136: TEST_network_ping_test2: ceph osd crush move rack3 datacenter=dc3 2026-02-25T22:24:37.808 INFO:tasks.workunit.client.0.trial188.stderr:no need to move item id -10 name 'rack3' to location {datacenter=dc3} in crush map 2026-02-25T22:24:37.819 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:137: TEST_network_ping_test2: ceph osd crush move host1 rack=rack1 2026-02-25T22:24:38.084 INFO:tasks.workunit.client.0.trial188.stderr:no need to move item id -11 name 'host1' to location {rack=rack1} in crush map 2026-02-25T22:24:38.094 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:138: TEST_network_ping_test2: ceph osd crush move host2 rack=rack2 2026-02-25T22:24:38.364 INFO:tasks.workunit.client.0.trial188.stderr:no need to move item id -12 name 'host2' to location {rack=rack2} in crush map 2026-02-25T22:24:38.375 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:139: TEST_network_ping_test2: ceph osd crush move host3 rack=rack3 2026-02-25T22:24:38.639 INFO:tasks.workunit.client.0.trial188.stderr:no need to move item id -13 name 'host3' to location {rack=rack3} in crush map 2026-02-25T22:24:38.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:140: TEST_network_ping_test2: ceph osd crush set osd.0 1.0 host=host1 2026-02-25T22:24:38.925 INFO:tasks.workunit.client.0.trial188.stderr:set item id 0 name 'osd.0' weight 1 at location {host=host1}: no change 2026-02-25T22:24:38.936 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:141: TEST_network_ping_test2: ceph osd crush set osd.1 1.0 host=host2 2026-02-25T22:24:39.203 INFO:tasks.workunit.client.0.trial188.stderr:set item id 1 name 'osd.1' weight 1 at location {host=host2}: no change 2026-02-25T22:24:39.213 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:142: TEST_network_ping_test2: ceph osd crush set osd.2 1.0 host=host3 2026-02-25T22:24:39.489 INFO:tasks.workunit.client.0.trial188.stderr:set item id 2 name 'osd.2' weight 1 at location {host=host3}: no change 2026-02-25T22:24:39.499 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:143: TEST_network_ping_test2: ceph osd crush rule create-simple myrule default host firstn 2026-02-25T22:24:39.776 INFO:tasks.workunit.client.0.trial188.stderr:rule myrule already exists 2026-02-25T22:24:39.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:145: TEST_network_ping_test2: create_pool foo 16 16 replicated myrule 2026-02-25T22:24:39.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create foo 16 16 replicated myrule 2026-02-25T22:24:40.058 INFO:tasks.workunit.client.0.trial188.stderr:pool 'foo' already exists 2026-02-25T22:24:40.068 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:24:41.070 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:148: TEST_network_ping_test2: timeout 20 rados bench -p foo 10 write -b 4096 --no-cleanup 2026-02-25T22:24:41.090 INFO:tasks.workunit.client.0.trial188.stdout:hints = 1 2026-02-25T22:24:41.091 INFO:tasks.workunit.client.0.trial188.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 10 seconds or 0 objects 2026-02-25T22:24:41.091 INFO:tasks.workunit.client.0.trial188.stdout:Object prefix: benchmark_data_trial188_35642 2026-02-25T22:24:41.091 INFO:tasks.workunit.client.0.trial188.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2026-02-25T22:24:41.091 INFO:tasks.workunit.client.0.trial188.stdout: 0 0 0 0 0 0 - 0 2026-02-25T22:24:42.090 INFO:tasks.workunit.client.0.trial188.stdout: 1 16 10851 10835 42.3209 42.3242 0.00140999 0.00147443 2026-02-25T22:24:43.091 INFO:tasks.workunit.client.0.trial188.stdout: 2 16 22785 22769 44.4673 46.6172 0.00127962 0.00140382 2026-02-25T22:24:44.091 INFO:tasks.workunit.client.0.trial188.stdout: 3 15 35734 35719 46.5056 50.5859 0.00137899 0.00134233 2026-02-25T22:24:45.091 INFO:tasks.workunit.client.0.trial188.stdout: 4 16 48109 48093 46.9622 48.3359 0.00108332 0.00132932 2026-02-25T22:24:46.091 INFO:tasks.workunit.client.0.trial188.stdout: 5 16 60581 60565 47.3127 48.7188 0.0008308 0.00131955 2026-02-25T22:24:47.091 INFO:tasks.workunit.client.0.trial188.stdout: 6 16 72443 72427 47.1494 46.3359 0.000923483 0.00132416 2026-02-25T22:24:48.091 INFO:tasks.workunit.client.0.trial188.stdout: 7 16 83490 83474 46.5779 43.1523 0.00171759 0.0013404 2026-02-25T22:24:49.091 INFO:tasks.workunit.client.0.trial188.stdout: 8 16 93495 93479 45.6405 39.082 0.00131865 0.00136799 2026-02-25T22:24:50.091 INFO:tasks.workunit.client.0.trial188.stdout: 9 16 103251 103235 44.8034 38.1094 0.00143388 0.00139355 2026-02-25T22:24:51.091 INFO:tasks.workunit.client.0.trial188.stdout: 10 3 113028 113025 44.147 38.2422 0.00151585 0.00141436 2026-02-25T22:24:52.091 INFO:tasks.workunit.client.0.trial188.stdout:Total time run: 10.001 2026-02-25T22:24:52.092 INFO:tasks.workunit.client.0.trial188.stdout:Total writes made: 113028 2026-02-25T22:24:52.092 INFO:tasks.workunit.client.0.trial188.stdout:Write size: 4096 2026-02-25T22:24:52.092 INFO:tasks.workunit.client.0.trial188.stdout:Object size: 4096 2026-02-25T22:24:52.092 INFO:tasks.workunit.client.0.trial188.stdout:Bandwidth (MB/sec): 44.147 2026-02-25T22:24:52.092 INFO:tasks.workunit.client.0.trial188.stdout:Stddev Bandwidth: 4.61984 2026-02-25T22:24:52.092 INFO:tasks.workunit.client.0.trial188.stdout:Max bandwidth (MB/sec): 50.5859 2026-02-25T22:24:52.092 INFO:tasks.workunit.client.0.trial188.stdout:Min bandwidth (MB/sec): 38.1094 2026-02-25T22:24:52.092 INFO:tasks.workunit.client.0.trial188.stdout:Average IOPS: 11301 2026-02-25T22:24:52.092 INFO:tasks.workunit.client.0.trial188.stdout:Stddev IOPS: 1182.68 2026-02-25T22:24:52.092 INFO:tasks.workunit.client.0.trial188.stdout:Max IOPS: 12950 2026-02-25T22:24:52.093 INFO:tasks.workunit.client.0.trial188.stdout:Min IOPS: 9756 2026-02-25T22:24:52.093 INFO:tasks.workunit.client.0.trial188.stdout:Average Latency(s): 0.00141436 2026-02-25T22:24:52.093 INFO:tasks.workunit.client.0.trial188.stdout:Stddev Latency(s): 0.000707769 2026-02-25T22:24:52.093 INFO:tasks.workunit.client.0.trial188.stdout:Max latency(s): 0.00970256 2026-02-25T22:24:52.093 INFO:tasks.workunit.client.0.trial188.stdout:Min latency(s): 0.000439885 2026-02-25T22:24:52.097 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:151: TEST_network_ping_test2: sleep 10 2026-02-25T22:25:02.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:152: TEST_network_ping_test2: flush_pg_stats 2026-02-25T22:25:02.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:25:02.100 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:25:02.319 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:25:02.319 INFO:tasks.workunit.client.0.trial188.stderr:1 2026-02-25T22:25:02.319 INFO:tasks.workunit.client.0.trial188.stderr:2' 2026-02-25T22:25:02.319 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:25:02.319 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:02.320 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:25:02.397 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-25T22:25:02.397 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-25T22:25:02.397 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491' 2026-02-25T22:25:02.397 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:02.398 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:25:02.482 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705674 2026-02-25T22:25:02.482 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705674 2026-02-25T22:25:02.482 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-38654705674' 2026-02-25T22:25:02.482 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:02.482 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:25:02.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=55834574858 2026-02-25T22:25:02.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 55834574858 2026-02-25T22:25:02.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-38654705674 2-55834574858' 2026-02-25T22:25:02.563 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:02.563 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836491 2026-02-25T22:25:02.563 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:02.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:25:02.565 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-25T22:25:02.565 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:02.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836491 2026-02-25T22:25:02.566 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836491 2026-02-25T22:25:02.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-25T22:25:02.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:25:02.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836489 -lt 21474836491 2026-02-25T22:25:02.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:25:03.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:25:03.787 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:25:04.004 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-25T22:25:04.004 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:04.005 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705674 2026-02-25T22:25:04.005 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:04.006 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:25:04.007 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705674 2026-02-25T22:25:04.007 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:04.008 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705674 2026-02-25T22:25:04.009 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.1 seq 38654705674 2026-02-25T22:25:04.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705674' 2026-02-25T22:25:04.009 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:25:04.225 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705674 -lt 38654705674 2026-02-25T22:25:04.225 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:04.225 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-55834574858 2026-02-25T22:25:04.225 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:04.226 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:25:04.227 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-55834574858 2026-02-25T22:25:04.227 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:04.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=55834574858 2026-02-25T22:25:04.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 55834574858' 2026-02-25T22:25:04.229 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.2 seq 55834574858 2026-02-25T22:25:04.229 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:25:04.445 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 55834574859 -lt 55834574858 2026-02-25T22:25:04.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: ceph health 2026-02-25T22:25:04.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:154: TEST_network_ping_test2: tee td/network-ping/health 2026-02-25T22:25:04.696 INFO:tasks.workunit.client.0.trial188.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.255ms); Slow OSD heartbeats on front (longest 0.245ms) 2026-02-25T22:25:04.707 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:155: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats' td/network-ping/health 2026-02-25T22:25:04.709 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: ceph health detail 2026-02-25T22:25:04.709 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:157: TEST_network_ping_test2: tee td/network-ping/health 2026-02-25T22:25:04.986 INFO:tasks.workunit.client.0.trial188.stdout:HEALTH_WARN Slow OSD heartbeats on back (longest 0.255ms); Slow OSD heartbeats on front (longest 0.245ms) 2026-02-25T22:25:04.986 INFO:tasks.workunit.client.0.trial188.stdout:[WRN] OSD_SLOW_PING_TIME_BACK: Slow OSD heartbeats on back (longest 0.255ms) 2026-02-25T22:25:04.986 INFO:tasks.workunit.client.0.trial188.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.255 msec 2026-02-25T22:25:04.987 INFO:tasks.workunit.client.0.trial188.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.226 msec 2026-02-25T22:25:04.987 INFO:tasks.workunit.client.0.trial188.stdout: Slow OSD heartbeats on back from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.183 msec 2026-02-25T22:25:04.987 INFO:tasks.workunit.client.0.trial188.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.163 msec 2026-02-25T22:25:04.987 INFO:tasks.workunit.client.0.trial188.stdout: Slow OSD heartbeats on back from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.155 msec 2026-02-25T22:25:04.987 INFO:tasks.workunit.client.0.trial188.stdout: Slow OSD heartbeats on back from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.154 msec possibly improving 2026-02-25T22:25:04.987 INFO:tasks.workunit.client.0.trial188.stdout:[WRN] OSD_SLOW_PING_TIME_FRONT: Slow OSD heartbeats on front (longest 0.245ms) 2026-02-25T22:25:04.987 INFO:tasks.workunit.client.0.trial188.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.2 [dc3,rack3] 0.245 msec 2026-02-25T22:25:04.987 INFO:tasks.workunit.client.0.trial188.stdout: Slow OSD heartbeats on front from osd.0 [dc1,rack1] to osd.1 [dc2,rack2] 0.207 msec 2026-02-25T22:25:04.987 INFO:tasks.workunit.client.0.trial188.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.1 [dc2,rack2] 0.198 msec possibly improving 2026-02-25T22:25:04.987 INFO:tasks.workunit.client.0.trial188.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.0 [dc1,rack1] 0.196 msec 2026-02-25T22:25:04.987 INFO:tasks.workunit.client.0.trial188.stdout: Slow OSD heartbeats on front from osd.2 [dc3,rack3] to osd.0 [dc1,rack1] 0.168 msec 2026-02-25T22:25:04.987 INFO:tasks.workunit.client.0.trial188.stdout: Slow OSD heartbeats on front from osd.1 [dc2,rack2] to osd.2 [dc3,rack3] 0.157 msec 2026-02-25T22:25:04.995 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:158: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_BACK td/network-ping/health 2026-02-25T22:25:04.996 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:159: TEST_network_ping_test2: grep -q OSD_SLOW_PING_TIME_FRONT td/network-ping/health 2026-02-25T22:25:04.997 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:160: TEST_network_ping_test2: grep -q 'Slow OSD heartbeats on front from osd[.][0-2] [[]dc[1-3],rack[1-3][]] to osd[.][0-2] [[]dc[1-3],rack[1-3][]]' td/network-ping/health 2026-02-25T22:25:04.999 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:162: TEST_network_ping_test2: rm -f td/network-ping/health 2026-02-25T22:25:05.000 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/network-ping.sh:20: run: teardown td/network-ping 2026-02-25T22:25:05.000 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-25T22:25:05.000 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:25:05.000 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-25T22:25:05.001 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:05.001 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:05.001 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:05.001 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:05.001 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:05.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:05.312 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:25:05.313 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:25:05.314 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:25:05.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:25:05.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:25:05.316 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:25:05.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:05.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:25:05.318 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:25:05.318 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:05.319 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:25:05.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:25:05.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-25T22:25:06.026 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:25:06.026 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:06.026 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:25:06.027 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.24105 2026-02-25T22:25:06.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:25:06.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:25:06.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T22:25:06.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/network-ping 0 2026-02-25T22:25:06.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/network-ping 2026-02-25T22:25:06.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T22:25:06.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/network-ping KILL 2026-02-25T22:25:06.028 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:06.029 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:06.029 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:06.029 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:06.029 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:06.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:06.032 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:25:06.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:25:06.033 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:25:06.034 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:25:06.034 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:25:06.034 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:25:06.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:06.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:25:06.036 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:25:06.036 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:06.037 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:25:06.038 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T22:25:06.038 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/network-ping 2026-02-25T22:25:06.039 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:25:06.039 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:06.039 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.24105 2026-02-25T22:25:06.039 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.24105 2026-02-25T22:25:06.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:25:06.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:25:06.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T22:25:06.041 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T22:25:06.041 DEBUG:teuthology.orchestra.run.trial188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T22:25:06.091 INFO:tasks.workunit:Running workunit misc/ok-to-stop.sh... 2026-02-25T22:25:06.091 DEBUG:teuthology.orchestra.run.trial188:workunit test misc/ok-to-stop.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61d4431d3ec879bd98659388b3655acf3d392418 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh 2026-02-25T22:25:06.144 INFO:tasks.workunit.client.0.trial188.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/ok-to-stop 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:6: run: local dir=td/ok-to-stop 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:7: run: shift 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: export CEPH_MON_A=127.0.0.1:7150 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:9: run: CEPH_MON_A=127.0.0.1:7150 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: export CEPH_MON_B=127.0.0.1:7151 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:10: run: CEPH_MON_B=127.0.0.1:7151 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: export CEPH_MON_C=127.0.0.1:7152 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:11: run: CEPH_MON_C=127.0.0.1:7152 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: export CEPH_MON_D=127.0.0.1:7153 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:12: run: CEPH_MON_D=127.0.0.1:7153 2026-02-25T22:25:06.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: export CEPH_MON_E=127.0.0.1:7154 2026-02-25T22:25:06.149 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:13: run: CEPH_MON_E=127.0.0.1:7154 2026-02-25T22:25:06.149 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:14: run: export CEPH_ARGS 2026-02-25T22:25:06.149 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: uuidgen 2026-02-25T22:25:06.149 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:15: run: CEPH_ARGS+='--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none ' 2026-02-25T22:25:06.149 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: export 'ORIG_CEPH_ARGS=--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none ' 2026-02-25T22:25:06.149 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:16: run: ORIG_CEPH_ARGS='--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none ' 2026-02-25T22:25:06.150 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: set 2026-02-25T22:25:06.150 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-25T22:25:06.152 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:18: run: local 'funcs=TEST_0_mds 2026-02-25T22:25:06.152 INFO:tasks.workunit.client.0.trial188.stderr:TEST_0_osd 2026-02-25T22:25:06.152 INFO:tasks.workunit.client.0.trial188.stderr:TEST_1_mon_checks 2026-02-25T22:25:06.152 INFO:tasks.workunit.client.0.trial188.stderr:TEST_2_mons_checks 2026-02-25T22:25:06.152 INFO:tasks.workunit.client.0.trial188.stderr:TEST_3_mons_checks 2026-02-25T22:25:06.152 INFO:tasks.workunit.client.0.trial188.stderr:TEST_4_mons_checks 2026-02-25T22:25:06.152 INFO:tasks.workunit.client.0.trial188.stderr:TEST_5_mons_checks' 2026-02-25T22:25:06.152 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:25:06.152 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:25:06.152 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:25:06.152 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:25:06.152 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:25:06.153 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:25:06.153 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:25:06.153 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:06.153 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:06.153 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:06.153 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:06.153 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:06.155 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:06.155 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:25:06.156 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:25:06.156 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:25:06.157 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:25:06.157 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:25:06.157 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:25:06.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:06.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:25:06.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:25:06.159 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:06.160 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:25:06.162 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:25:06.162 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:25:06.163 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:25:06.163 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:06.163 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:06.163 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:25:06.164 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:25:06.165 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:25:06.165 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:25:06.166 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:25:06.166 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:06.166 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:06.166 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35988 2026-02-25T22:25:06.167 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:25:06.169 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T22:25:06.169 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T22:25:06.169 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:25:06.169 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:25:06.169 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_mds td/ok-to-stop 2026-02-25T22:25:06.169 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:208: TEST_0_mds: local dir=td/ok-to-stop 2026-02-25T22:25:06.169 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:210: TEST_0_mds: CEPH_ARGS='--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:25:06.169 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:212: TEST_0_mds: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:25:06.169 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:25:06.170 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:25:06.170 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:25:06.170 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:25:06.170 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:25:06.170 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:25:06.190 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:25:06.190 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:06.190 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:06.191 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:06.191 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:06.191 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:06.191 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:06.191 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:25:06.218 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:25:06.219 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:25:06.219 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:25:06.219 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:25:06.220 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:25:06.221 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:25:06.221 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:25:06.221 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:25:06.221 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:25:06.221 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:06.221 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:06.221 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:06.221 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:25:06.221 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:25:06.222 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get fsid 2026-02-25T22:25:06.277 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:25:06.278 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:25:06.278 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:25:06.278 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:25:06.278 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:25:06.279 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:25:06.279 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:25:06.279 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:25:06.279 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:06.279 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:06.280 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:06.280 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:25:06.280 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:25:06.280 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get mon_host 2026-02-25T22:25:06.335 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:213: TEST_0_mds: run_mgr td/ok-to-stop x 2026-02-25T22:25:06.336 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-stop 2026-02-25T22:25:06.336 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:25:06.336 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:25:06.336 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:25:06.336 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-stop/x 2026-02-25T22:25:06.336 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:25:06.450 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:25:06.450 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:06.450 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:06.450 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:06.450 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:06.450 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:06.451 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:06.451 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:25:06.451 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:25:06.471 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:214: TEST_0_mds: run_osd td/ok-to-stop 0 2026-02-25T22:25:06.471 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-25T22:25:06.471 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:25:06.471 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:25:06.471 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:25:06.471 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/0 2026-02-25T22:25:06.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:25:06.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:25:06.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:25:06.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:25:06.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2026-02-25T22:25:06.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2026-02-25T22:25:06.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:25:06.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:25:06.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-25T22:25:06.472 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:25:06.472 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:06.472 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:06.472 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:06.473 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:06.473 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:06.473 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:06.473 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:06.473 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:25:06.473 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:25:06.473 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:25:06.473 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-25T22:25:06.473 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:25:06.473 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:25:06.473 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:25:06.473 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:25:06.474 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:25:06.474 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:25:06.474 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:25:06.474 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/0 2026-02-25T22:25:06.474 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:25:06.475 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=b91d6dc0-a572-40de-9b5f-d67c4eb34d2c 2026-02-25T22:25:06.475 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 b91d6dc0-a572-40de-9b5f-d67c4eb34d2c' 2026-02-25T22:25:06.475 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 b91d6dc0-a572-40de-9b5f-d67c4eb34d2c 2026-02-25T22:25:06.475 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:25:06.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDCdp9p20OsHBAAS+A85xyNoN6ucFN2RD3AAQ== 2026-02-25T22:25:06.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDCdp9p20OsHBAAS+A85xyNoN6ucFN2RD3AAQ=="}' 2026-02-25T22:25:06.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new b91d6dc0-a572-40de-9b5f-d67c4eb34d2c -i td/ok-to-stop/0/new.json 2026-02-25T22:25:06.592 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:25:06.600 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/0/new.json 2026-02-25T22:25:06.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDCdp9p20OsHBAAS+A85xyNoN6ucFN2RD3AAQ== --osd-uuid b91d6dc0-a572-40de-9b5f-d67c4eb34d2c 2026-02-25T22:25:06.616 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:06.610+0000 7f0faa27aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:06.618 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:06.610+0000 7f0faa27aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:06.618 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:06.610+0000 7f0faa27aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:06.619 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:06.610+0000 7f0faa27aa40 -1 bdev(0x5595c0012400 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:25:06.619 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:06.610+0000 7f0faa27aa40 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2026-02-25T22:25:07.283 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/0/keyring 2026-02-25T22:25:07.283 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:25:07.284 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:25:07.284 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:25:07.284 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:25:07.408 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:25:07.408 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:25:07.409 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:25:07.409 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:25:07.409 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:25:07.409 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:25:07.429 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:07.422+0000 7f3bfe4e7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:07.431 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:07.422+0000 7f3bfe4e7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:07.432 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:07.426+0000 7f3bfe4e7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:07.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:25:07.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:25:07.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:25:07.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:25:07.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:25:07.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:07.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:25:07.518 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:25:07.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:07.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:25:07.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:07.825 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:07.818+0000 7f3bfe4e7a40 -1 Falling back to public interface 2026-02-25T22:25:08.602 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:08.594+0000 7f3bfe4e7a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:25:08.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:08.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:08.630 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:25:08.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:25:08.632 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:25:08.635 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:08.778 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:09.779 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:09.780 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:09.780 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:25:09.780 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:25:09.780 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:09.780 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:25:09.997 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2354223031,v1:127.0.0.1:6801/2354223031] [v2:127.0.0.1:6802/2354223031,v1:127.0.0.1:6803/2354223031] exists,up b91d6dc0-a572-40de-9b5f-d67c4eb34d2c 2026-02-25T22:25:09.998 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:25:09.998 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:25:09.998 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:25:09.998 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:215: TEST_0_mds: run_mds td/ok-to-stop a 2026-02-25T22:25:09.998 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ok-to-stop 2026-02-25T22:25:09.998 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2026-02-25T22:25:09.998 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=a 2026-02-25T22:25:09.998 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2026-02-25T22:25:09.998 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ok-to-stop/a 2026-02-25T22:25:09.998 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2026-02-25T22:25:09.998 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:09.998 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:09.999 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:09.999 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:09.999 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:09.999 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:09.999 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id a --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:25:10.018 INFO:tasks.workunit.client.0.trial188.stdout:starting mds.a at 2026-02-25T22:25:10.021 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:217: TEST_0_mds: ceph osd pool create meta 1 2026-02-25T22:25:10.301 INFO:tasks.workunit.client.0.trial188.stderr:pool 'meta' already exists 2026-02-25T22:25:10.310 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:218: TEST_0_mds: ceph osd pool create data 1 2026-02-25T22:25:10.603 INFO:tasks.workunit.client.0.trial188.stderr:pool 'data' already exists 2026-02-25T22:25:10.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:219: TEST_0_mds: ceph fs new myfs meta data 2026-02-25T22:25:10.961 INFO:tasks.workunit.client.0.trial188.stderr:filesystem 'myfs' already exists 2026-02-25T22:25:10.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:220: TEST_0_mds: sleep 5 2026-02-25T22:25:15.972 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:222: TEST_0_mds: ceph mds ok-to-stop a 2026-02-25T22:25:16.164 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4123 and/or standby gids 2026-02-25T22:25:16.169 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:223: TEST_0_mds: ceph mds ok-to-stop a dne 2026-02-25T22:25:16.362 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4123 and/or standby gids 2026-02-25T22:25:16.367 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:224: TEST_0_mds: ceph mds ok-to-stop dne 2026-02-25T22:25:16.649 INFO:tasks.workunit.client.0.trial188.stderr:should be safe to stop [dne] 2026-02-25T22:25:16.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:226: TEST_0_mds: run_mds td/ok-to-stop b 2026-02-25T22:25:16.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:570: run_mds: local dir=td/ok-to-stop 2026-02-25T22:25:16.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:571: run_mds: shift 2026-02-25T22:25:16.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:572: run_mds: local id=b 2026-02-25T22:25:16.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:573: run_mds: shift 2026-02-25T22:25:16.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:574: run_mds: local data=td/ok-to-stop/b 2026-02-25T22:25:16.660 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: get_asok_path 2026-02-25T22:25:16.660 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:16.660 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:16.660 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:16.660 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:16.660 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:16.660 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:16.661 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:577: run_mds: ceph-mds --id b --debug-mds 20 --debug-objecter 20 --debug-ms 20 --chdir= --mds-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:25:16.677 INFO:tasks.workunit.client.0.trial188.stdout:starting mds.b at 2026-02-25T22:25:16.680 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:227: TEST_0_mds: sleep 5 2026-02-25T22:25:21.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:229: TEST_0_mds: ceph mds ok-to-stop a 2026-02-25T22:25:21.960 INFO:tasks.workunit.client.0.trial188.stderr:should be safe to stop [a] 2026-02-25T22:25:21.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:230: TEST_0_mds: ceph mds ok-to-stop b 2026-02-25T22:25:22.264 INFO:tasks.workunit.client.0.trial188.stderr:should be safe to stop [b] 2026-02-25T22:25:22.273 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:231: TEST_0_mds: ceph mds ok-to-stop a b 2026-02-25T22:25:22.474 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4123 and/or standby gids 4137 2026-02-25T22:25:22.477 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:232: TEST_0_mds: ceph mds ok-to-stop a dne1 dne2 2026-02-25T22:25:22.756 INFO:tasks.workunit.client.0.trial188.stderr:should be safe to stop [a,dne1,dne2] 2026-02-25T22:25:22.765 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:233: TEST_0_mds: ceph mds ok-to-stop b dne 2026-02-25T22:25:23.059 INFO:tasks.workunit.client.0.trial188.stderr:should be safe to stop [b,dne] 2026-02-25T22:25:23.068 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:234: TEST_0_mds: ceph mds ok-to-stop a b dne 2026-02-25T22:25:23.271 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: insufficent standby MDS daemons to stop active gids 4123 and/or standby gids 4137 2026-02-25T22:25:23.275 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:235: TEST_0_mds: ceph mds ok-to-stop dne1 dne2 2026-02-25T22:25:23.554 INFO:tasks.workunit.client.0.trial188.stderr:should be safe to stop [dne1,dne2] 2026-02-25T22:25:23.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:237: TEST_0_mds: kill_daemons td/ok-to-stop KILL mds.a 2026-02-25T22:25:23.565 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:23.565 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:23.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:23.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:23.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:23.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:23.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:25:23.673 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:23.673 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:23.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:23.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:23.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:23.783 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:23.783 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:25:23.783 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:25:23.783 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:25:23.783 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:25:23.784 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:23.784 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:23.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:23.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:23.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:23.789 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:23.789 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:25:23.790 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:25:23.790 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:25:23.791 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:25:23.791 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:25:23.791 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:25:23.792 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:23.792 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:25:23.793 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:25:23.793 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:23.794 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:25:23.795 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:25:23.795 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:25:23.801 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:25:23.801 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:23.801 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:23.801 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:25:23.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:25:23.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:25:23.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:25:23.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:25:23.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:25:23.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:25:23.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:25:23.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:25:23.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:25:23.804 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:23.804 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:23.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:23.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:23.805 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:23.806 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:23.806 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:25:23.807 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:25:23.807 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:25:23.808 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:25:23.808 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:25:23.808 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:25:23.809 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:23.809 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:25:23.810 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:25:23.810 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:23.811 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:25:23.812 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:25:23.812 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:25:23.813 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:25:23.813 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:23.813 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:23.813 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:25:23.814 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:25:23.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:25:23.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:25:23.816 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:25:23.816 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:23.816 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:23.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35988 2026-02-25T22:25:23.817 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:25:23.819 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:25:23.819 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:25:23.819 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:25:23.819 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_0_osd td/ok-to-stop 2026-02-25T22:25:23.819 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:241: TEST_0_osd: local dir=td/ok-to-stop 2026-02-25T22:25:23.819 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:243: TEST_0_osd: CEPH_ARGS='--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:25:23.819 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:245: TEST_0_osd: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:25:23.819 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:25:23.820 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:25:23.820 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:25:23.820 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:25:23.820 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:25:23.820 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:25:23.839 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:25:23.839 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:23.839 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:23.840 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:23.840 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:23.840 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:23.840 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:23.841 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:25:23.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:25:23.870 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:25:23.870 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:25:23.870 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:25:23.871 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:25:23.872 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:25:23.872 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:25:23.872 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:25:23.872 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:25:23.872 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:23.872 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:23.872 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:23.872 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:25:23.872 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:25:23.873 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get fsid 2026-02-25T22:25:23.929 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:25:23.929 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:25:23.929 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:25:23.929 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:25:23.929 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:25:23.930 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:25:23.931 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:25:23.931 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:25:23.931 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:23.931 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:23.931 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:23.931 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:25:23.931 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:25:23.931 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get mon_host 2026-02-25T22:25:23.987 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:246: TEST_0_osd: run_mgr td/ok-to-stop x 2026-02-25T22:25:23.987 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ok-to-stop 2026-02-25T22:25:23.987 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:25:23.987 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:25:23.987 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:25:23.987 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ok-to-stop/x 2026-02-25T22:25:23.987 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:25:24.102 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:25:24.102 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:24.102 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:24.102 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:24.102 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:24.102 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:24.103 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:24.103 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:25:24.103 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ok-to-stop/x '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:25:24.123 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:247: TEST_0_osd: run_osd td/ok-to-stop 0 2026-02-25T22:25:24.123 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-25T22:25:24.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:25:24.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:25:24.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:25:24.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/0 2026-02-25T22:25:24.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:25:24.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:25:24.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:25:24.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:25:24.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/0' 2026-02-25T22:25:24.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/0/journal' 2026-02-25T22:25:24.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:25:24.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:25:24.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-25T22:25:24.125 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:25:24.125 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:24.125 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:24.125 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:24.125 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:24.125 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:24.125 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:24.126 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:24.126 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:25:24.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:25:24.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:25:24.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-25T22:25:24.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:25:24.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:25:24.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:25:24.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:25:24.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:25:24.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:25:24.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:25:24.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/0 2026-02-25T22:25:24.127 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:25:24.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4048d5e5-97f9-434d-978b-5c70ae144b17 2026-02-25T22:25:24.128 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 4048d5e5-97f9-434d-978b-5c70ae144b17 2026-02-25T22:25:24.128 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 4048d5e5-97f9-434d-978b-5c70ae144b17' 2026-02-25T22:25:24.128 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:25:24.136 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDUdp9pVxP5BxAAjSGeZK5J5SV0+PYAbzB+xg== 2026-02-25T22:25:24.136 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDUdp9pVxP5BxAAjSGeZK5J5SV0+PYAbzB+xg=="}' 2026-02-25T22:25:24.136 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4048d5e5-97f9-434d-978b-5c70ae144b17 -i td/ok-to-stop/0/new.json 2026-02-25T22:25:24.246 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:25:24.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/0/new.json 2026-02-25T22:25:24.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDUdp9pVxP5BxAAjSGeZK5J5SV0+PYAbzB+xg== --osd-uuid 4048d5e5-97f9-434d-978b-5c70ae144b17 2026-02-25T22:25:24.269 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:24.262+0000 7f838d7b3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:24.271 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:24.262+0000 7f838d7b3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:24.272 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:24.266+0000 7f838d7b3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:24.272 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:24.266+0000 7f838d7b3a40 -1 bdev(0x562540c92400 td/ok-to-stop/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:25:24.272 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:24.266+0000 7f838d7b3a40 -1 bluestore(td/ok-to-stop/0) _read_fsid unparsable uuid 2026-02-25T22:25:24.967 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/0/keyring 2026-02-25T22:25:24.967 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:25:24.968 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:25:24.968 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:25:24.969 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:25:25.094 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:25:25.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:25:25.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/0 --osd-journal=td/ok-to-stop/0/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:25:25.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:25:25.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:25:25.096 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:25:25.116 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:25.110+0000 7f6814b6da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:25.118 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:25.110+0000 7f6814b6da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:25.119 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:25.110+0000 7f6814b6da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:25.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:25:25.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:25:25.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:25:25.207 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:25:25.207 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:25:25.207 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:25.207 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:25:25.208 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:25:25.208 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:25.208 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:25:25.319 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:25.665 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:25.658+0000 7f6814b6da40 -1 Falling back to public interface 2026-02-25T22:25:26.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:26.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:26.321 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:25:26.321 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:25:26.324 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:26.324 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:25:26.435 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:26.555 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:26.550+0000 7f6814b6da40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:25:27.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:27.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:27.437 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:25:27.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:25:27.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:27.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:25:27.656 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2815671937,v1:127.0.0.1:6801/2815671937] [v2:127.0.0.1:6802/2815671937,v1:127.0.0.1:6803/2815671937] exists,up 4048d5e5-97f9-434d-978b-5c70ae144b17 2026-02-25T22:25:27.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:25:27.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:25:27.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:25:27.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:248: TEST_0_osd: run_osd td/ok-to-stop 1 2026-02-25T22:25:27.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-25T22:25:27.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:25:27.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:25:27.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:25:27.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/1 2026-02-25T22:25:27.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:25:27.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:25:27.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:25:27.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:25:27.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/1' 2026-02-25T22:25:27.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/1/journal' 2026-02-25T22:25:27.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:25:27.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:25:27.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-25T22:25:27.660 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:25:27.660 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:27.661 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:27.661 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:27.661 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:27.661 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:27.661 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:27.661 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:27.661 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:25:27.661 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:25:27.661 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:25:27.661 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-25T22:25:27.661 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:25:27.661 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:25:27.662 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:25:27.662 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:25:27.662 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:25:27.662 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:25:27.662 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:25:27.662 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/1 2026-02-25T22:25:27.662 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:25:27.663 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=3cae4c45-378a-4a59-bff0-7f714c3b1992 2026-02-25T22:25:27.663 INFO:tasks.workunit.client.0.trial188.stdout:add osd1 3cae4c45-378a-4a59-bff0-7f714c3b1992 2026-02-25T22:25:27.663 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 3cae4c45-378a-4a59-bff0-7f714c3b1992' 2026-02-25T22:25:27.663 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:25:27.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDXdp9phcf2JxAAWgg6+JzqJZaupeumn4Mwbg== 2026-02-25T22:25:27.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDXdp9phcf2JxAAWgg6+JzqJZaupeumn4Mwbg=="}' 2026-02-25T22:25:27.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 3cae4c45-378a-4a59-bff0-7f714c3b1992 -i td/ok-to-stop/1/new.json 2026-02-25T22:25:27.900 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:25:27.910 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/1/new.json 2026-02-25T22:25:27.910 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDXdp9phcf2JxAAWgg6+JzqJZaupeumn4Mwbg== --osd-uuid 3cae4c45-378a-4a59-bff0-7f714c3b1992 2026-02-25T22:25:27.927 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:27.918+0000 7f4def39fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:27.928 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:27.922+0000 7f4def39fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:27.929 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:27.922+0000 7f4def39fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:27.930 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:27.922+0000 7f4def39fa40 -1 bdev(0x558f6ded0400 td/ok-to-stop/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:25:27.930 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:27.922+0000 7f4def39fa40 -1 bluestore(td/ok-to-stop/1) _read_fsid unparsable uuid 2026-02-25T22:25:28.578 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/1/keyring 2026-02-25T22:25:28.578 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:25:28.579 INFO:tasks.workunit.client.0.trial188.stdout:adding osd1 key to auth repository 2026-02-25T22:25:28.579 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:25:28.579 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:25:28.876 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:25:28.876 INFO:tasks.workunit.client.0.trial188.stdout:start osd.1 2026-02-25T22:25:28.877 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/1 --osd-journal=td/ok-to-stop/1/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:25:28.877 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:25:28.877 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:25:28.877 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:25:28.898 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:28.890+0000 7f1c81ac8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:28.900 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:28.894+0000 7f1c81ac8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:28.901 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:28.894+0000 7f1c81ac8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:29.086 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:25:29.086 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:25:29.086 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:25:29.086 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:25:29.086 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:25:29.086 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:29.087 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:25:29.087 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:25:29.087 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:29.087 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:25:29.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:29.304 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:29.298+0000 7f1c81ac8a40 -1 Falling back to public interface 2026-02-25T22:25:29.948 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:29.942+0000 7f1c81ac8a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:25:30.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:30.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:30.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:25:30.305 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:25:30.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:30.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:25:30.515 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:31.517 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:31.517 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:31.517 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:25:31.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:25:31.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:31.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:25:31.734 INFO:tasks.workunit.client.0.trial188.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/1977198390,v1:127.0.0.1:6811/1977198390] [v2:127.0.0.1:6812/1977198390,v1:127.0.0.1:6813/1977198390] exists,up 3cae4c45-378a-4a59-bff0-7f714c3b1992 2026-02-25T22:25:31.737 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:25:31.737 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:25:31.737 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:25:31.737 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:249: TEST_0_osd: run_osd td/ok-to-stop 2 2026-02-25T22:25:31.737 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-25T22:25:31.737 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:25:31.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:25:31.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:25:31.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/2 2026-02-25T22:25:31.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:25:31.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:25:31.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:25:31.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:25:31.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/2' 2026-02-25T22:25:31.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/2/journal' 2026-02-25T22:25:31.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:25:31.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:25:31.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-25T22:25:31.739 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:25:31.739 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:31.739 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:31.739 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:31.739 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:31.739 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:31.739 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:31.739 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:31.739 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:25:31.739 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:25:31.739 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:25:31.739 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-25T22:25:31.739 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:25:31.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:25:31.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:25:31.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:25:31.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:25:31.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:25:31.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:25:31.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/2 2026-02-25T22:25:31.741 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:25:31.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=5904c4b9-2ff6-4a5a-bd12-bfc0403fc112 2026-02-25T22:25:31.741 INFO:tasks.workunit.client.0.trial188.stdout:add osd2 5904c4b9-2ff6-4a5a-bd12-bfc0403fc112 2026-02-25T22:25:31.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 5904c4b9-2ff6-4a5a-bd12-bfc0403fc112' 2026-02-25T22:25:31.742 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:25:31.749 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDbdp9pBqeHLBAA8neLypemqiZx7LpJhVMDOQ== 2026-02-25T22:25:31.750 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDbdp9pBqeHLBAA8neLypemqiZx7LpJhVMDOQ=="}' 2026-02-25T22:25:31.750 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 5904c4b9-2ff6-4a5a-bd12-bfc0403fc112 -i td/ok-to-stop/2/new.json 2026-02-25T22:25:31.973 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:25:31.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/2/new.json 2026-02-25T22:25:31.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDbdp9pBqeHLBAA8neLypemqiZx7LpJhVMDOQ== --osd-uuid 5904c4b9-2ff6-4a5a-bd12-bfc0403fc112 2026-02-25T22:25:32.008 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:32.002+0000 7fdccedd9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:32.009 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:32.002+0000 7fdccedd9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:32.010 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:32.002+0000 7fdccedd9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:32.011 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:32.002+0000 7fdccedd9a40 -1 bdev(0x55d98ba6e400 td/ok-to-stop/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:25:32.011 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:32.002+0000 7fdccedd9a40 -1 bluestore(td/ok-to-stop/2) _read_fsid unparsable uuid 2026-02-25T22:25:32.678 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/2/keyring 2026-02-25T22:25:32.679 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:25:32.679 INFO:tasks.workunit.client.0.trial188.stdout:adding osd2 key to auth repository 2026-02-25T22:25:32.680 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:25:32.680 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:25:32.969 INFO:tasks.workunit.client.0.trial188.stdout:start osd.2 2026-02-25T22:25:32.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:25:32.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/2 --osd-journal=td/ok-to-stop/2/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:25:32.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:25:32.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:25:32.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:25:32.991 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:32.986+0000 7f9bb787da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:32.993 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:32.986+0000 7f9bb787da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:32.994 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:32.986+0000 7f9bb787da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:33.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:25:33.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:25:33.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:25:33.187 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:25:33.187 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:25:33.187 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:33.187 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:25:33.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:25:33.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:33.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:25:33.365 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:33.358+0000 7f9bb787da40 -1 Falling back to public interface 2026-02-25T22:25:33.407 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:33.936 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:33.930+0000 7f9bb787da40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:25:34.409 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:34.409 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:34.409 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:25:34.410 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:25:34.410 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:34.410 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:25:34.616 INFO:tasks.workunit.client.0.trial188.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1918013958,v1:127.0.0.1:6819/1918013958] [v2:127.0.0.1:6820/1918013958,v1:127.0.0.1:6821/1918013958] exists,up 5904c4b9-2ff6-4a5a-bd12-bfc0403fc112 2026-02-25T22:25:34.618 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:25:34.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:25:34.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:25:34.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:250: TEST_0_osd: run_osd td/ok-to-stop 3 2026-02-25T22:25:34.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ok-to-stop 2026-02-25T22:25:34.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:25:34.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-25T22:25:34.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:25:34.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ok-to-stop/3 2026-02-25T22:25:34.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:25:34.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:25:34.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:25:34.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:25:34.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ok-to-stop/3' 2026-02-25T22:25:34.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ok-to-stop/3/journal' 2026-02-25T22:25:34.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:25:34.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:25:34.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ok-to-stop' 2026-02-25T22:25:34.620 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:25:34.620 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:34.620 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:34.620 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:34.620 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:34.620 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:34.621 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:34.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:34.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:25:34.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:25:34.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:25:34.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ok-to-stop/$name.log' 2026-02-25T22:25:34.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ok-to-stop/$name.pid' 2026-02-25T22:25:34.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:25:34.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:25:34.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:25:34.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:25:34.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:25:34.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:25:34.622 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ok-to-stop/3 2026-02-25T22:25:34.622 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:25:34.622 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d0d29ebb-b75b-4a82-bd84-ae06c766cac8 2026-02-25T22:25:34.623 INFO:tasks.workunit.client.0.trial188.stdout:add osd3 d0d29ebb-b75b-4a82-bd84-ae06c766cac8 2026-02-25T22:25:34.623 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 d0d29ebb-b75b-4a82-bd84-ae06c766cac8' 2026-02-25T22:25:34.623 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:25:34.632 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDedp9pp5WLJRAAy7R6osrLnMbyaEm0mo3JQQ== 2026-02-25T22:25:34.632 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDedp9pp5WLJRAAy7R6osrLnMbyaEm0mo3JQQ=="}' 2026-02-25T22:25:34.632 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d0d29ebb-b75b-4a82-bd84-ae06c766cac8 -i td/ok-to-stop/3/new.json 2026-02-25T22:25:34.856 INFO:tasks.workunit.client.0.trial188.stdout:3 2026-02-25T22:25:34.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ok-to-stop/3/new.json 2026-02-25T22:25:34.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDedp9pp5WLJRAAy7R6osrLnMbyaEm0mo3JQQ== --osd-uuid d0d29ebb-b75b-4a82-bd84-ae06c766cac8 2026-02-25T22:25:34.891 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:34.886+0000 7f428ce26a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:34.893 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:34.886+0000 7f428ce26a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:34.894 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:34.886+0000 7f428ce26a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:34.895 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:34.886+0000 7f428ce26a40 -1 bdev(0x5579136b6400 td/ok-to-stop/3/block) open stat got: (1) Operation not permitted 2026-02-25T22:25:34.895 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:34.886+0000 7f428ce26a40 -1 bluestore(td/ok-to-stop/3) _read_fsid unparsable uuid 2026-02-25T22:25:35.570 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ok-to-stop/3/keyring 2026-02-25T22:25:35.571 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:25:35.572 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-25T22:25:35.572 INFO:tasks.workunit.client.0.trial188.stdout:adding osd3 key to auth repository 2026-02-25T22:25:35.572 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ok-to-stop/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:25:35.871 INFO:tasks.workunit.client.0.trial188.stdout:start osd.3 2026-02-25T22:25:35.872 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-25T22:25:35.872 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ok-to-stop/3 --osd-journal=td/ok-to-stop/3/journal --chdir= --run-dir=td/ok-to-stop '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ok-to-stop/$name.log' '--pid-file=td/ok-to-stop/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:25:35.872 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:25:35.872 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:25:35.872 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:25:35.885 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:35.878+0000 7f7156547a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:35.886 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:35.878+0000 7f7156547a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:35.887 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:35.882+0000 7f7156547a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:25:36.088 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-25T22:25:36.088 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:25:36.088 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-25T22:25:36.088 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:25:36.089 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:25:36.089 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:36.089 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:25:36.089 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:25:36.090 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:36.090 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:25:36.233 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:36.226+0000 7f7156547a40 -1 Falling back to public interface 2026-02-25T22:25:36.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:25:36.825 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:25:36.818+0000 7f7156547a40 -1 osd.3 0 log_to_monitors true 2026-02-25T22:25:37.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:25:37.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:25:37.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:25:37.306 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:25:37.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:25:37.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-25T22:25:37.519 INFO:tasks.workunit.client.0.trial188.stdout:osd.3 up in weight 1 up_from 19 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2289042172,v1:127.0.0.1:6827/2289042172] [v2:127.0.0.1:6828/2289042172,v1:127.0.0.1:6829/2289042172] exists,up d0d29ebb-b75b-4a82-bd84-ae06c766cac8 2026-02-25T22:25:37.520 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:25:37.520 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:25:37.520 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:25:37.520 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:252: TEST_0_osd: ceph osd erasure-code-profile set ec-profile m=2 k=2 crush-failure-domain=osd 2026-02-25T22:25:37.797 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:253: TEST_0_osd: ceph osd pool create ec erasure ec-profile 2026-02-25T22:25:38.189 INFO:tasks.workunit.client.0.trial188.stderr:pool 'ec' already exists 2026-02-25T22:25:38.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:255: TEST_0_osd: wait_for_clean 2026-02-25T22:25:38.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:25:38.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:25:38.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:25:38.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:25:38.202 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:25:38.202 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:25:38.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:25:38.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:25:38.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:25:38.308 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:25:38.308 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:25:38.308 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:25:38.308 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:25:38.309 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:25:38.309 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:25:38.532 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:25:38.532 INFO:tasks.workunit.client.0.trial188.stderr:1 2026-02-25T22:25:38.532 INFO:tasks.workunit.client.0.trial188.stderr:2 2026-02-25T22:25:38.533 INFO:tasks.workunit.client.0.trial188.stderr:3' 2026-02-25T22:25:38.533 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:25:38.533 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:38.533 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:25:38.609 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-25T22:25:38.609 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-25T22:25:38.610 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485' 2026-02-25T22:25:38.610 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:38.610 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:25:38.690 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672964 2026-02-25T22:25:38.696 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672964 2026-02-25T22:25:38.696 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485 1-42949672964' 2026-02-25T22:25:38.696 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:38.697 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:25:38.768 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542147 2026-02-25T22:25:38.768 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542147 2026-02-25T22:25:38.768 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485 1-42949672964 2-60129542147' 2026-02-25T22:25:38.768 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:38.768 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:25:38.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378627 2026-02-25T22:25:38.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378627 2026-02-25T22:25:38.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485 1-42949672964 2-60129542147 3-81604378627' 2026-02-25T22:25:38.851 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:38.851 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836485 2026-02-25T22:25:38.851 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:38.852 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:25:38.853 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-25T22:25:38.853 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:38.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836485 2026-02-25T22:25:38.854 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836485 2026-02-25T22:25:38.855 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-25T22:25:38.855 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:25:39.081 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836485 2026-02-25T22:25:39.082 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:25:40.083 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:25:40.083 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:25:40.294 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836485 2026-02-25T22:25:40.295 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:25:41.296 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:25:41.296 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:25:41.513 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-25T22:25:41.513 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:41.513 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672964 2026-02-25T22:25:41.513 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:41.515 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:25:41.515 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672964 2026-02-25T22:25:41.515 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:41.517 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672964 2026-02-25T22:25:41.517 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.1 seq 42949672964 2026-02-25T22:25:41.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672964' 2026-02-25T22:25:41.518 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:25:41.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672964 -lt 42949672964 2026-02-25T22:25:41.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:41.739 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-60129542147 2026-02-25T22:25:41.739 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:41.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:25:41.741 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542147 2026-02-25T22:25:41.741 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:41.742 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=60129542147 2026-02-25T22:25:41.743 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.2 seq 60129542147 2026-02-25T22:25:41.743 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 60129542147' 2026-02-25T22:25:41.743 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:25:41.958 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 60129542148 -lt 60129542147 2026-02-25T22:25:41.958 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:41.958 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-81604378627 2026-02-25T22:25:41.958 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:41.959 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:25:41.960 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378627 2026-02-25T22:25:41.960 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:41.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378627 2026-02-25T22:25:41.962 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.3 seq 81604378627 2026-02-25T22:25:41.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 81604378627' 2026-02-25T22:25:41.963 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:25:42.179 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378627 -lt 81604378627 2026-02-25T22:25:42.180 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:25:42.180 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:25:42.181 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:25:42.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-25T22:25:42.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:25:42.461 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:25:42.461 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:25:42.461 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:25:42.462 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:25:42.462 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:25:42.462 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:25:42.669 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-25T22:25:42.669 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:25:42.670 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:25:42.670 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:25:42.946 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-25T22:25:42.947 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:25:42.947 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:25:42.947 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:258: TEST_0_osd: ceph osd pool set ec min_size 3 2026-02-25T22:25:43.206 INFO:tasks.workunit.client.0.trial188.stderr:set pool 1 min_size to 3 2026-02-25T22:25:43.218 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:259: TEST_0_osd: wait_for_clean 2026-02-25T22:25:43.218 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:25:43.219 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:25:43.219 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:25:43.219 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:25:43.220 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:25:43.220 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:25:43.220 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:25:43.220 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:25:43.220 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:25:43.334 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:25:43.334 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:25:43.334 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:25:43.334 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:25:43.334 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:25:43.334 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:25:43.546 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:25:43.546 INFO:tasks.workunit.client.0.trial188.stderr:1 2026-02-25T22:25:43.546 INFO:tasks.workunit.client.0.trial188.stderr:2 2026-02-25T22:25:43.546 INFO:tasks.workunit.client.0.trial188.stderr:3' 2026-02-25T22:25:43.546 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:25:43.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:43.547 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:25:43.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836488 2026-02-25T22:25:43.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836488 2026-02-25T22:25:43.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488' 2026-02-25T22:25:43.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:43.630 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:25:43.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672967 2026-02-25T22:25:43.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672967 2026-02-25T22:25:43.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-42949672967' 2026-02-25T22:25:43.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:43.711 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:25:43.789 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542150 2026-02-25T22:25:43.790 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542150 2026-02-25T22:25:43.790 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-60129542150' 2026-02-25T22:25:43.790 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:43.790 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:25:43.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378630 2026-02-25T22:25:43.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378630 2026-02-25T22:25:43.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836488 1-42949672967 2-60129542150 3-81604378630' 2026-02-25T22:25:43.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:43.868 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836488 2026-02-25T22:25:43.869 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:43.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:25:43.870 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836488 2026-02-25T22:25:43.870 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:43.871 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836488 2026-02-25T22:25:43.872 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836488 2026-02-25T22:25:43.872 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836488' 2026-02-25T22:25:43.872 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:25:44.080 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836485 -lt 21474836488 2026-02-25T22:25:44.080 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:25:45.081 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:25:45.081 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:25:45.297 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836488 -lt 21474836488 2026-02-25T22:25:45.297 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:45.298 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672967 2026-02-25T22:25:45.298 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:45.299 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:25:45.300 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672967 2026-02-25T22:25:45.300 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:45.301 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672967 2026-02-25T22:25:45.301 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.1 seq 42949672967 2026-02-25T22:25:45.302 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672967' 2026-02-25T22:25:45.302 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:25:45.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672967 -lt 42949672967 2026-02-25T22:25:45.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:45.519 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-60129542150 2026-02-25T22:25:45.519 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:45.520 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:25:45.520 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542150 2026-02-25T22:25:45.520 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:45.522 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=60129542150 2026-02-25T22:25:45.522 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.2 seq 60129542150 2026-02-25T22:25:45.522 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 60129542150' 2026-02-25T22:25:45.523 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:25:45.745 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 60129542150 -lt 60129542150 2026-02-25T22:25:45.746 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:45.746 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-81604378630 2026-02-25T22:25:45.746 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:45.747 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:25:45.748 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378630 2026-02-25T22:25:45.748 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:45.749 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378630 2026-02-25T22:25:45.750 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.3 seq 81604378630 2026-02-25T22:25:45.750 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 81604378630' 2026-02-25T22:25:45.750 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:25:45.977 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378630 -lt 81604378630 2026-02-25T22:25:45.978 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:25:45.978 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:25:45.978 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:25:46.269 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-25T22:25:46.269 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:25:46.269 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:25:46.269 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:25:46.269 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:25:46.269 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:25:46.270 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:25:46.270 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:25:46.486 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-25T22:25:46.486 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:25:46.487 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:25:46.487 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:25:46.779 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-25T22:25:46.779 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:25:46.779 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:25:46.779 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:261: TEST_0_osd: ceph osd ok-to-stop 0 2026-02-25T22:25:46.981 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:46.991 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:262: TEST_0_osd: ceph osd ok-to-stop 1 2026-02-25T22:25:47.206 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:47.215 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:263: TEST_0_osd: ceph osd ok-to-stop 2 2026-02-25T22:25:47.429 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:47.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:264: TEST_0_osd: ceph osd ok-to-stop 3 2026-02-25T22:25:47.657 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:47.666 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:265: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-25T22:25:47.834 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":false,"osds":[0,1],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:47.835 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-25T22:25:47.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:266: TEST_0_osd: ceph osd ok-to-stop 2 3 2026-02-25T22:25:47.987 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":false,"osds":[2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:47.988 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-25T22:25:47.993 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2026-02-25T22:25:47.993 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:267: TEST_0_osd: grep '[0]' 2026-02-25T22:25:48.208 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:48.209 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: ceph osd ok-to-stop 1 --max 2 2026-02-25T22:25:48.209 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:268: TEST_0_osd: grep '[1]' 2026-02-25T22:25:48.428 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:48.429 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:271: TEST_0_osd: ceph osd pool set ec min_size 2 2026-02-25T22:25:48.735 INFO:tasks.workunit.client.0.trial188.stderr:set pool 1 min_size to 2 2026-02-25T22:25:48.752 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:272: TEST_0_osd: wait_for_clean 2026-02-25T22:25:48.753 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:25:48.753 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:25:48.753 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:25:48.753 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:25:48.754 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:25:48.754 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:25:48.754 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:25:48.754 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:25:48.754 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:25:48.863 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:25:48.863 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:25:48.863 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:25:48.863 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:25:48.863 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:25:48.863 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:25:49.086 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:25:49.086 INFO:tasks.workunit.client.0.trial188.stderr:1 2026-02-25T22:25:49.086 INFO:tasks.workunit.client.0.trial188.stderr:2 2026-02-25T22:25:49.086 INFO:tasks.workunit.client.0.trial188.stderr:3' 2026-02-25T22:25:49.086 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:25:49.086 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:49.087 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:25:49.164 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836491 2026-02-25T22:25:49.164 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836491 2026-02-25T22:25:49.165 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491' 2026-02-25T22:25:49.165 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:49.165 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:25:49.244 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672970 2026-02-25T22:25:49.244 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672970 2026-02-25T22:25:49.244 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970' 2026-02-25T22:25:49.245 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:49.245 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:25:49.323 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542153 2026-02-25T22:25:49.323 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542153 2026-02-25T22:25:49.323 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153' 2026-02-25T22:25:49.324 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:49.324 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:25:49.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378633 2026-02-25T22:25:49.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378633 2026-02-25T22:25:49.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836491 1-42949672970 2-60129542153 3-81604378633' 2026-02-25T22:25:49.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:49.402 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836491 2026-02-25T22:25:49.402 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:49.402 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:25:49.403 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836491 2026-02-25T22:25:49.403 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:49.404 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836491 2026-02-25T22:25:49.404 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836491 2026-02-25T22:25:49.405 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836491' 2026-02-25T22:25:49.405 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:25:49.618 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836489 -lt 21474836491 2026-02-25T22:25:49.618 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:25:50.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:25:50.620 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:25:50.835 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836491 -lt 21474836491 2026-02-25T22:25:50.836 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:50.836 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672970 2026-02-25T22:25:50.836 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:50.837 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:25:50.838 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672970 2026-02-25T22:25:50.838 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:50.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672970 2026-02-25T22:25:50.840 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.1 seq 42949672970 2026-02-25T22:25:50.841 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672970' 2026-02-25T22:25:50.841 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:25:51.048 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672970 -lt 42949672970 2026-02-25T22:25:51.048 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:51.048 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-60129542153 2026-02-25T22:25:51.048 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:51.049 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:25:51.050 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542153 2026-02-25T22:25:51.050 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:51.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=60129542153 2026-02-25T22:25:51.052 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.2 seq 60129542153 2026-02-25T22:25:51.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 60129542153' 2026-02-25T22:25:51.053 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:25:51.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 60129542154 -lt 60129542153 2026-02-25T22:25:51.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:51.272 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-81604378633 2026-02-25T22:25:51.272 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:51.273 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:25:51.274 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378633 2026-02-25T22:25:51.274 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:51.275 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378633 2026-02-25T22:25:51.276 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.3 seq 81604378633 2026-02-25T22:25:51.276 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 81604378633' 2026-02-25T22:25:51.276 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:25:51.490 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378633 -lt 81604378633 2026-02-25T22:25:51.491 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:25:51.491 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:25:51.491 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:25:51.771 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 32 == 0 2026-02-25T22:25:51.771 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:25:51.771 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:25:51.771 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:25:51.771 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:25:51.771 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:25:51.772 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:25:51.772 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:25:51.977 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=32 2026-02-25T22:25:51.978 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:25:51.978 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:25:51.978 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:25:52.260 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 32 = 32 2026-02-25T22:25:52.260 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:25:52.261 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:25:52.261 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:274: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-25T22:25:52.466 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:52.476 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:275: TEST_0_osd: ceph osd ok-to-stop 2 3 2026-02-25T22:25:52.693 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[2,3],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:52.704 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:276: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2026-02-25T22:25:52.859 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:52.860 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-25T22:25:52.863 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:277: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2026-02-25T22:25:53.019 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:53.019 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-25T22:25:53.024 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: ceph osd ok-to-stop 0 --max 2 2026-02-25T22:25:53.025 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:279: TEST_0_osd: grep '[0,1]' 2026-02-25T22:25:53.241 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:53.241 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: ceph osd ok-to-stop 0 --max 20 2026-02-25T22:25:53.242 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:280: TEST_0_osd: grep '[0,1]' 2026-02-25T22:25:53.458 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:53.458 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: ceph osd ok-to-stop 2 --max 2 2026-02-25T22:25:53.458 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:281: TEST_0_osd: grep '[2,3]' 2026-02-25T22:25:53.672 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:53.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: ceph osd ok-to-stop 2 --max 20 2026-02-25T22:25:53.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:282: TEST_0_osd: grep '[2,3]' 2026-02-25T22:25:53.888 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[0,2],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:53.889 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:285: TEST_0_osd: kill_daemons td/ok-to-stop TERM osd.0 2026-02-25T22:25:53.890 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:53.890 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:53.890 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:53.890 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:53.890 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:54.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:54.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:286: TEST_0_osd: ceph osd down 0 2026-02-25T22:25:54.405 INFO:tasks.workunit.client.0.trial188.stderr:osd.0 is already down. 2026-02-25T22:25:54.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:287: TEST_0_osd: wait_for_peered 2026-02-25T22:25:54.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1731: wait_for_peered: local cmd= 2026-02-25T22:25:54.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1732: wait_for_peered: local num_peered=-1 2026-02-25T22:25:54.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1733: wait_for_peered: local cur_peered 2026-02-25T22:25:54.416 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: wait_for_peered: get_timeout_delays 90 .1 2026-02-25T22:25:54.416 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:25:54.417 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:25:54.417 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:25:54.417 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:25:54.417 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:25:54.521 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: wait_for_peered: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:25:54.521 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1734: wait_for_peered: local -a delays 2026-02-25T22:25:54.521 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1735: wait_for_peered: local -i loop=0 2026-02-25T22:25:54.521 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1737: wait_for_peered: flush_pg_stats 2026-02-25T22:25:54.522 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:25:54.522 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:25:54.729 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:25:54.729 INFO:tasks.workunit.client.0.trial188.stderr:1 2026-02-25T22:25:54.729 INFO:tasks.workunit.client.0.trial188.stderr:2 2026-02-25T22:25:54.730 INFO:tasks.workunit.client.0.trial188.stderr:3' 2026-02-25T22:25:54.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:25:54.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:54.730 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:25:54.792 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq= 2026-02-25T22:25:54.793 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: true 2026-02-25T22:25:54.793 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z '' 2026-02-25T22:25:54.793 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: continue 2026-02-25T22:25:54.793 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:54.793 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:25:54.875 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=42949672973 2026-02-25T22:25:54.875 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 42949672973 2026-02-25T22:25:54.875 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 1-42949672973' 2026-02-25T22:25:54.875 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:54.875 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.2 flush_pg_stats 2026-02-25T22:25:54.955 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=60129542156 2026-02-25T22:25:54.955 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 60129542156 2026-02-25T22:25:54.955 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 1-42949672973 2-60129542156' 2026-02-25T22:25:54.955 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:25:54.955 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.3 flush_pg_stats 2026-02-25T22:25:55.036 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=81604378636 2026-02-25T22:25:55.036 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 81604378636 2026-02-25T22:25:55.036 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 1-42949672973 2-60129542156 3-81604378636' 2026-02-25T22:25:55.036 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:55.036 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-42949672973 2026-02-25T22:25:55.037 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:55.037 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:25:55.038 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-42949672973 2026-02-25T22:25:55.038 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:55.039 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=42949672973 2026-02-25T22:25:55.040 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.1 seq 42949672973 2026-02-25T22:25:55.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 42949672973' 2026-02-25T22:25:55.040 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:25:55.260 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672971 -lt 42949672973 2026-02-25T22:25:55.260 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:25:56.261 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:25:56.262 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:25:56.479 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 42949672974 -lt 42949672973 2026-02-25T22:25:56.479 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:56.479 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 2-60129542156 2026-02-25T22:25:56.479 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:56.480 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=2 2026-02-25T22:25:56.481 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 2-60129542156 2026-02-25T22:25:56.481 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:56.482 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=60129542156 2026-02-25T22:25:56.483 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.2 seq 60129542156 2026-02-25T22:25:56.483 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.2 seq 60129542156' 2026-02-25T22:25:56.483 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 2 2026-02-25T22:25:56.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 60129542157 -lt 60129542156 2026-02-25T22:25:56.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:25:56.711 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 3-81604378636 2026-02-25T22:25:56.711 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:25:56.713 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=3 2026-02-25T22:25:56.713 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 3-81604378636 2026-02-25T22:25:56.713 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:25:56.715 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=81604378636 2026-02-25T22:25:56.715 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.3 seq 81604378636 2026-02-25T22:25:56.715 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.3 seq 81604378636' 2026-02-25T22:25:56.716 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 3 2026-02-25T22:25:56.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 81604378636 -lt 81604378636 2026-02-25T22:25:56.942 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: wait_for_peered: get_num_pgs 2026-02-25T22:25:56.942 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:25:56.942 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:25:57.227 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1738: wait_for_peered: test 32 == 0 2026-02-25T22:25:57.227 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1742: wait_for_peered: true 2026-02-25T22:25:57.227 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1746: wait_for_peered: get_num_active_or_peered 2026-02-25T22:25:57.228 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2026-02-25T22:25:57.228 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2026-02-25T22:25:57.228 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2026-02-25T22:25:57.228 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2026-02-25T22:25:57.228 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2026-02-25T22:25:57.442 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1746: wait_for_peered: cur_peered=32 2026-02-25T22:25:57.443 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: get_num_pgs 2026-02-25T22:25:57.443 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:25:57.443 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:25:57.734 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: test 32 = 32 2026-02-25T22:25:57.734 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1747: wait_for_peered: break 2026-02-25T22:25:57.734 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1762: wait_for_peered: return 0 2026-02-25T22:25:57.734 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:289: TEST_0_osd: ceph osd ok-to-stop 0 2026-02-25T22:25:57.939 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[0],"num_ok_pgs":0,"num_not_ok_pgs":0} 2026-02-25T22:25:57.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:290: TEST_0_osd: ceph osd ok-to-stop 0 1 2026-02-25T22:25:58.169 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":true,"osds":[0,1],"num_ok_pgs":32,"num_not_ok_pgs":0,"ok_become_degraded":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:58.180 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:291: TEST_0_osd: ceph osd ok-to-stop 0 1 2 2026-02-25T22:25:58.334 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":false,"osds":[0,1,2],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:58.334 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-25T22:25:58.338 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:292: TEST_0_osd: ceph osd ok-to-stop 1 2 3 2026-02-25T22:25:58.486 INFO:tasks.workunit.client.0.trial188.stdout:{"ok_to_stop":false,"osds":[1,2,3],"num_ok_pgs":0,"num_not_ok_pgs":32,"bad_become_inactive":["1.0","1.1","1.2","1.3","1.4","1.5","1.6","1.7","1.8","1.9","1.a","1.b","1.c","1.d","1.e","1.f","1.10","1.11","1.12","1.13","1.14","1.15","1.16","1.17","1.18","1.19","1.1a","1.1b","1.1c","1.1d","1.1e","1.1f"]} 2026-02-25T22:25:58.486 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: unsafe to stop osd(s) at this time (32 PGs are or would become offline) 2026-02-25T22:25:58.491 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:25:58.492 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:58.493 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:58.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:58.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:58.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:58.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:58.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:25:58.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:25:58.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:25:58.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:25:58.802 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:58.803 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:58.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:58.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:58.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:58.809 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:58.809 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:25:58.810 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:25:58.810 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:25:58.811 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:25:58.811 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:25:58.811 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:25:58.812 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:58.812 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:25:58.813 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:25:58.813 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:58.814 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:25:58.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:25:58.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:25:58.835 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:25:58.836 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:58.836 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:58.836 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:25:58.838 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:25:58.838 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:25:58.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:25:58.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:25:58.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:25:58.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:25:58.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:25:58.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:25:58.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:25:58.839 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:58.839 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:58.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:58.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:58.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:58.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:58.840 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:25:58.841 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:25:58.842 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:25:58.843 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:25:58.843 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:25:58.843 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:25:58.844 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:58.844 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:25:58.845 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:25:58.845 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:58.845 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:25:58.846 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:25:58.847 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:25:58.848 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:25:58.848 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:58.848 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:58.848 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:25:58.849 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:25:58.849 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:25:58.849 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:25:58.851 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:25:58.851 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:58.851 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:58.851 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35988 2026-02-25T22:25:58.852 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:25:58.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:25:58.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:25:58.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:25:58.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_1_mon_checks td/ok-to-stop 2026-02-25T22:25:58.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:28: TEST_1_mon_checks: local dir=td/ok-to-stop 2026-02-25T22:25:58.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:30: TEST_1_mon_checks: CEPH_ARGS='--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150 ' 2026-02-25T22:25:58.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:32: TEST_1_mon_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:25:58.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:25:58.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:25:58.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:25:58.855 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:25:58.855 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:25:58.855 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:25:58.875 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:25:58.875 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:58.875 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:58.875 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:58.875 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:58.875 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:58.876 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:58.876 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:25:58.902 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:25:58.903 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:25:58.903 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:25:58.903 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:25:58.903 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:25:58.904 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:25:58.904 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:25:58.904 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:25:58.904 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:25:58.904 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:58.905 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:58.905 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:58.905 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:25:58.905 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:25:58.905 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get fsid 2026-02-25T22:25:58.959 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:25:58.959 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:25:58.959 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:25:58.959 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:25:58.959 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:25:58.961 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:25:58.961 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:25:58.961 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:25:58.961 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:58.961 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:58.961 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:58.961 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:25:58.961 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:25:58.961 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get mon_host 2026-02-25T22:25:59.013 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:34: TEST_1_mon_checks: ceph mon ok-to-stop dne 2026-02-25T22:25:59.115 INFO:tasks.workunit.client.0.trial188.stderr:invalid mon(s) specified: [dne] 2026-02-25T22:25:59.122 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:35: TEST_1_mon_checks: ceph mon ok-to-stop a 2026-02-25T22:25:59.220 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a] 2026-02-25T22:25:59.224 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:37: TEST_1_mon_checks: ceph mon ok-to-add-offline 2026-02-25T22:25:59.307 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-25T22:25:59.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:39: TEST_1_mon_checks: ceph mon ok-to-rm a 2026-02-25T22:25:59.396 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: removing mon.a would break quorum 2026-02-25T22:25:59.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:40: TEST_1_mon_checks: ceph mon ok-to-rm dne 2026-02-25T22:25:59.504 INFO:tasks.workunit.client.0.trial188.stderr:mon.dne does not exist 2026-02-25T22:25:59.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:25:59.513 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:59.513 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:59.513 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:59.513 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:59.514 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:59.622 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:59.622 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:25:59.622 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:25:59.622 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:25:59.622 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:25:59.622 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:59.622 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:59.623 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:59.623 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:59.623 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:59.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:59.628 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:25:59.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:25:59.629 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:25:59.631 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:25:59.631 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:25:59.631 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:25:59.632 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:59.632 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:25:59.633 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:25:59.633 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:59.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:25:59.635 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:25:59.635 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:25:59.638 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:25:59.638 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:59.638 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:59.638 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:25:59.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:25:59.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:25:59.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:25:59.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:25:59.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:25:59.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:25:59.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:25:59.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:25:59.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:25:59.641 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:25:59.641 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:25:59.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:25:59.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:25:59.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:25:59.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:25:59.642 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:25:59.643 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:25:59.644 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:25:59.645 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:25:59.645 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:25:59.646 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:25:59.647 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:59.647 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:25:59.647 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:25:59.648 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:25:59.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:25:59.649 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:25:59.649 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:25:59.651 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:25:59.651 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:59.651 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:59.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:25:59.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:25:59.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:25:59.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:25:59.653 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:25:59.653 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:59.653 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:59.653 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35988 2026-02-25T22:25:59.655 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:25:59.656 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:25:59.656 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:25:59.656 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:25:59.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_2_mons_checks td/ok-to-stop 2026-02-25T22:25:59.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:44: TEST_2_mons_checks: local dir=td/ok-to-stop 2026-02-25T22:25:59.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:46: TEST_2_mons_checks: CEPH_ARGS='--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151 ' 2026-02-25T22:25:59.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:48: TEST_2_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:25:59.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:25:59.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:25:59.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:25:59.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:25:59.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:25:59.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:25:59.678 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:25:59.678 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:59.678 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:59.678 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:59.678 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:59.678 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:59.679 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:59.679 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:25:59.707 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:25:59.709 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:25:59.709 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:25:59.709 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:25:59.709 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:25:59.709 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:25:59.709 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:25:59.709 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:25:59.709 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:25:59.710 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:59.710 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:59.711 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:59.711 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:25:59.711 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:25:59.711 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get fsid 2026-02-25T22:25:59.765 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:25:59.765 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:25:59.765 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:25:59.766 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:25:59.767 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:25:59.767 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:25:59.767 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:25:59.767 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:25:59.767 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:59.767 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:59.767 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:59.767 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:25:59.768 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:25:59.768 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get mon_host 2026-02-25T22:25:59.821 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:49: TEST_2_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-25T22:25:59.821 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:25:59.821 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:25:59.821 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-25T22:25:59.821 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:25:59.822 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-25T22:25:59.822 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-25T22:25:59.841 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:25:59.841 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:25:59.841 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:25:59.841 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:25:59.841 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:59.841 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:59.842 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:25:59.842 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-25T22:25:59.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:25:59.870 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-25T22:25:59.871 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:25:59.871 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:25:59.871 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:25:59.872 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:25:59.872 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:25:59.872 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:25:59.872 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:25:59.872 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:59.872 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:59.872 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:59.872 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.b.asok 2026-02-25T22:25:59.873 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:25:59.873 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.b.asok config get fsid 2026-02-25T22:25:59.927 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-25T22:25:59.927 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:25:59.927 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:25:59.927 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:25:59.928 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:25:59.928 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:25:59.928 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:25:59.929 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:25:59.929 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:25:59.929 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:25:59.929 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:25:59.929 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.b.asok 2026-02-25T22:25:59.929 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:25:59.929 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.b.asok config get mon_host 2026-02-25T22:25:59.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:51: TEST_2_mons_checks: ceph mon ok-to-stop dne 2026-02-25T22:26:00.090 INFO:tasks.workunit.client.0.trial188.stderr:invalid mon(s) specified: [dne] 2026-02-25T22:26:00.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:52: TEST_2_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:26:00.183 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (b) after stopping mons [a] 2026-02-25T22:26:00.187 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:53: TEST_2_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:26:00.272 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b] 2026-02-25T22:26:00.274 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:54: TEST_2_mons_checks: ceph mon ok-to-stop a b 2026-02-25T22:26:00.373 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b] 2026-02-25T22:26:00.375 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:56: TEST_2_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:26:00.491 INFO:tasks.workunit.client.0.trial188.stderr:adding another mon that is not yet online will not break quorum 2026-02-25T22:26:00.499 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:58: TEST_2_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:26:00.614 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.a 2026-02-25T22:26:00.622 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:59: TEST_2_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:26:00.735 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.b 2026-02-25T22:26:00.743 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:60: TEST_2_mons_checks: ceph mon ok-to-rm dne 2026-02-25T22:26:00.844 INFO:tasks.workunit.client.0.trial188.stderr:mon.dne does not exist 2026-02-25T22:26:00.852 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:26:00.853 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:26:00.853 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:26:00.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:26:00.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:26:00.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:26:00.961 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:26:00.961 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:26:00.961 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:26:00.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:26:00.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:26:00.962 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:26:00.962 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:26:00.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:26:00.963 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:26:00.963 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:26:00.968 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:26:00.969 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:26:00.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:26:00.970 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:26:00.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:26:00.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:26:00.972 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:26:00.972 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:26:00.973 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:26:00.973 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:26:00.973 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:26:00.974 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:26:00.975 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:26:00.975 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:26:00.978 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:26:00.978 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:00.978 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:00.978 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:26:00.980 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:26:00.980 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:26:00.980 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:26:00.980 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:26:00.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:26:00.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:26:00.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:26:00.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:26:00.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:26:00.981 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:26:00.981 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:26:00.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:26:00.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:26:00.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:26:00.982 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:26:00.983 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:26:00.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:26:00.984 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:26:00.984 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:26:00.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:26:00.985 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:26:00.986 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:26:00.986 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:26:00.986 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:26:00.987 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:26:00.988 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:26:00.989 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:26:00.989 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:26:00.990 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:26:00.990 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:00.990 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:00.990 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:26:00.991 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:26:00.992 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:26:00.992 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:26:00.993 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:26:00.993 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:00.993 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:00.993 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35988 2026-02-25T22:26:00.995 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:26:00.996 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:26:00.996 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:26:00.996 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:26:00.996 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_3_mons_checks td/ok-to-stop 2026-02-25T22:26:00.996 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:64: TEST_3_mons_checks: local dir=td/ok-to-stop 2026-02-25T22:26:00.997 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:66: TEST_3_mons_checks: CEPH_ARGS='--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152 ' 2026-02-25T22:26:00.997 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:68: TEST_3_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:26:00.997 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:26:00.997 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:26:00.997 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:26:00.997 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:26:00.997 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:26:00.997 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:26:01.017 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:26:01.017 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:01.017 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:01.018 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:01.018 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:01.018 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:01.019 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:26:01.019 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:26:01.046 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:26:01.047 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:26:01.047 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:01.047 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:26:01.048 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:26:01.048 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:26:01.049 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:26:01.049 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:26:01.049 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:26:01.049 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:01.049 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:01.049 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:01.049 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:26:01.049 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:01.049 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get fsid 2026-02-25T22:26:01.104 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:26:01.104 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:01.104 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:26:01.104 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:26:01.105 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:26:01.105 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:26:01.105 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:26:01.105 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:26:01.105 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:01.105 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:01.105 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:01.106 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:26:01.106 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:01.106 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get mon_host 2026-02-25T22:26:01.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:69: TEST_3_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-25T22:26:01.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:26:01.160 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:26:01.160 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-25T22:26:01.160 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:26:01.160 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-25T22:26:01.160 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-25T22:26:01.179 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:26:01.179 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:01.179 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:01.180 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:01.180 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:01.180 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:01.181 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:26:01.181 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-25T22:26:01.208 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:26:01.209 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-25T22:26:01.209 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:01.209 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:26:01.209 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:26:01.210 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:26:01.210 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:26:01.210 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:26:01.210 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:26:01.211 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:01.211 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:01.211 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:01.211 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.b.asok 2026-02-25T22:26:01.211 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:01.211 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.b.asok config get fsid 2026-02-25T22:26:01.266 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-25T22:26:01.266 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:01.266 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:26:01.266 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:26:01.267 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:26:01.267 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:26:01.267 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:26:01.267 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:26:01.267 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:01.268 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:01.268 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:01.268 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.b.asok 2026-02-25T22:26:01.268 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:01.268 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.b.asok config get mon_host 2026-02-25T22:26:01.321 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:70: TEST_3_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-25T22:26:01.322 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:26:01.322 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:26:01.322 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-25T22:26:01.322 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:26:01.322 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2026-02-25T22:26:01.322 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-25T22:26:01.341 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:26:01.341 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:01.341 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:01.341 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:01.341 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:01.341 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:01.342 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:26:01.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-25T22:26:01.367 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:26:01.368 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-25T22:26:01.368 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:01.368 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-25T22:26:01.368 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:26:01.369 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:26:01.369 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-25T22:26:01.369 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-25T22:26:01.369 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-25T22:26:01.369 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:01.370 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:01.370 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:01.370 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.c.asok 2026-02-25T22:26:01.370 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:01.370 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.c.asok config get fsid 2026-02-25T22:26:01.425 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-25T22:26:01.425 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:01.425 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-25T22:26:01.425 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:26:01.425 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:26:01.425 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-25T22:26:01.425 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-25T22:26:01.426 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-25T22:26:01.426 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:01.426 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:01.426 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:01.426 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.c.asok 2026-02-25T22:26:01.426 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:01.426 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.c.asok config get mon_host 2026-02-25T22:26:01.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:71: TEST_3_mons_checks: wait_for_quorum 60 3 2026-02-25T22:26:01.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:26:01.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-25T22:26:01.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:26:01.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-25T22:26:01.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:26:01.485 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:26:01.485 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772058421 2026-02-25T22:26:01.485 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:01.486 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058361 -lt 1772058421 ]] 2026-02-25T22:26:01.486 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:26:01.486 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:01.805 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:01.805 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:01.805 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:01.806 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:26:01.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:26:01.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:26:01.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:26:01.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:26:01.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:26:01.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:73: TEST_3_mons_checks: ceph mon ok-to-stop dne 2026-02-25T22:26:01.918 INFO:tasks.workunit.client.0.trial188.stderr:invalid mon(s) specified: [dne] 2026-02-25T22:26:01.929 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:74: TEST_3_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:26:02.033 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (b,c) after stopping [a] 2026-02-25T22:26:02.043 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:75: TEST_3_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:26:02.150 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (a,c) after stopping [b] 2026-02-25T22:26:02.158 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:76: TEST_3_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:26:02.279 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (a,b) after stopping [c] 2026-02-25T22:26:02.287 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:77: TEST_3_mons_checks: ceph mon ok-to-stop a b 2026-02-25T22:26:02.388 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a,b] 2026-02-25T22:26:02.390 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:78: TEST_3_mons_checks: ceph mon ok-to-stop b c 2026-02-25T22:26:02.475 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [b,c] 2026-02-25T22:26:02.477 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:79: TEST_3_mons_checks: ceph mon ok-to-stop a b c 2026-02-25T22:26:02.565 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available () after stopping mons [a,b,c] 2026-02-25T22:26:02.569 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:81: TEST_3_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:26:02.674 INFO:tasks.workunit.client.0.trial188.stderr:adding another mon that is not yet online will not break quorum 2026-02-25T22:26:02.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:83: TEST_3_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:26:02.804 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.a 2026-02-25T22:26:02.812 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:84: TEST_3_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:26:02.916 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.b 2026-02-25T22:26:02.924 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:85: TEST_3_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:26:03.041 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.c 2026-02-25T22:26:03.051 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:87: TEST_3_mons_checks: kill_daemons td/ok-to-stop KILL mon.b 2026-02-25T22:26:03.052 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:26:03.052 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:26:03.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:26:03.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:26:03.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:26:03.160 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:26:03.160 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:88: TEST_3_mons_checks: wait_for_quorum 60 2 2026-02-25T22:26:03.160 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:26:03.161 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=2 2026-02-25T22:26:03.161 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:26:03.161 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 2 ]] 2026-02-25T22:26:03.161 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:26:03.161 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:26:03.162 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772058423 2026-02-25T22:26:03.162 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:03.162 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058363 -lt 1772058423 ]] 2026-02-25T22:26:03.163 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:26:03.163 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:03.279 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:03.279 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:03.280 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:03.280 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:26:03.289 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:03.289 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:03.289 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:03.290 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058363 -lt 1772058423 ]] 2026-02-25T22:26:03.290 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:26:03.291 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:03.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:03.401 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:03.402 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:26:03.402 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:03.412 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:03.412 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:03.412 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:03.413 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058363 -lt 1772058423 ]] 2026-02-25T22:26:03.413 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:26:03.414 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:06.531 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:06.531 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:06.532 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:06.532 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:26:06.541 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:06.541 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:06.541 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:06.542 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058366 -lt 1772058423 ]] 2026-02-25T22:26:06.542 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:26:06.543 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:06.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:06.659 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:06.659 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:26:06.659 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":4,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:06.668 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:06.668 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:06.669 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:06.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058366 -lt 1772058423 ]] 2026-02-25T22:26:06.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:26:06.670 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:06.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:06.784 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:06.785 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:26:06.785 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":5,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:06.795 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:06.795 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:06.795 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:06.796 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058366 -lt 1772058423 ]] 2026-02-25T22:26:06.797 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:26:06.797 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:09.914 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:09.914 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:09.915 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":4,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:09.915 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:26:09.924 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:09.924 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:09.924 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:09.925 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058369 -lt 1772058423 ]] 2026-02-25T22:26:09.925 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-25T22:26:09.926 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:19.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:19.040 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:19.041 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:01.008187Z","created":"2026-02-25T22:26:01.008187Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008182+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008184+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:01:008185+0000","crush_location":[]}]}}' 2026-02-25T22:26:19.041 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-25T22:26:19.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:26:19.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:26:19.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:26:19.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:26:19.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:26:19.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:90: TEST_3_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:26:22.138 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (c) after stopping mons [a] 2026-02-25T22:26:22.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:91: TEST_3_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:26:22.246 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (a,c) after stopping [b] 2026-02-25T22:26:22.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:92: TEST_3_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:26:25.353 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (a) after stopping mons [c] 2026-02-25T22:26:25.356 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:94: TEST_3_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:26:25.440 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-25T22:26:25.444 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:96: TEST_3_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:26:25.529 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: removing mon.a would break quorum 2026-02-25T22:26:25.532 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:97: TEST_3_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:26:25.634 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.b 2026-02-25T22:26:25.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:98: TEST_3_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:26:28.731 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: removing mon.c would break quorum 2026-02-25T22:26:28.735 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:26:28.736 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:26:28.736 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:26:28.736 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:26:28.736 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:26:28.736 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:26:28.845 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:26:28.845 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:26:28.845 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:26:28.845 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:26:28.845 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:26:28.845 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:26:28.845 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:26:28.846 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:26:28.846 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:26:28.846 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:26:28.851 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:26:28.851 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:26:28.852 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:26:28.853 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:26:28.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:26:28.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:26:28.855 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:26:28.856 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:26:28.856 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:26:28.856 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:26:28.856 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:26:28.857 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:26:28.858 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:26:28.858 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:26:28.864 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:26:28.864 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:28.865 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:28.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:26:28.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:26:28.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:26:28.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:26:28.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:26:28.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:26:28.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:26:28.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:26:28.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:26:28.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:26:28.868 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:26:28.868 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:26:28.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:26:28.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:26:28.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:26:28.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:26:28.869 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:26:28.870 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:26:28.871 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:26:28.872 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:26:28.872 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:26:28.872 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:26:28.873 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:26:28.873 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:26:28.874 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:26:28.874 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:26:28.875 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:26:28.876 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:26:28.876 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:26:28.877 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:26:28.877 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:28.878 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:28.878 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:26:28.879 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:26:28.879 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:26:28.879 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:26:28.880 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:26:28.880 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:28.880 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:28.881 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35988 2026-02-25T22:26:28.882 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:26:28.883 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:26:28.883 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:26:28.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:26:28.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_4_mons_checks td/ok-to-stop 2026-02-25T22:26:28.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:102: TEST_4_mons_checks: local dir=td/ok-to-stop 2026-02-25T22:26:28.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:104: TEST_4_mons_checks: CEPH_ARGS='--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153 ' 2026-02-25T22:26:28.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:106: TEST_4_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:26:28.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:26:28.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:26:28.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:26:28.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:26:28.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:26:28.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:26:28.903 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:26:28.903 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:28.903 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:28.904 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:28.904 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:28.904 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:28.904 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:26:28.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:26:28.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:26:28.939 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:26:28.939 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:28.939 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:26:28.939 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:26:28.940 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:26:28.940 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:26:28.941 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:26:28.941 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:26:28.941 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:28.941 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:28.941 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:28.941 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:26:28.941 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:28.941 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get fsid 2026-02-25T22:26:28.996 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:26:28.996 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:28.997 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:26:28.997 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:26:28.998 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:26:28.998 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:26:28.998 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:26:28.998 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:26:28.998 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:28.998 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:28.998 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:28.998 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:26:28.999 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:28.999 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get mon_host 2026-02-25T22:26:29.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:107: TEST_4_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-25T22:26:29.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:26:29.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:26:29.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-25T22:26:29.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:26:29.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-25T22:26:29.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-25T22:26:29.072 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:26:29.072 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:29.072 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:29.073 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:29.073 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:29.073 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:29.073 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:26:29.073 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-25T22:26:29.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:26:29.099 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-25T22:26:29.099 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:29.099 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:26:29.099 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:26:29.100 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:26:29.100 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:26:29.101 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:26:29.101 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:26:29.101 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:29.101 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:29.101 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:29.101 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.b.asok 2026-02-25T22:26:29.102 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:29.102 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.b.asok config get fsid 2026-02-25T22:26:29.156 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-25T22:26:29.156 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:29.156 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:26:29.157 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:26:29.158 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:26:29.158 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:26:29.158 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:26:29.158 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:26:29.158 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:29.158 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:29.158 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:29.158 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.b.asok 2026-02-25T22:26:29.159 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:29.159 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.b.asok config get mon_host 2026-02-25T22:26:29.213 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:108: TEST_4_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-25T22:26:29.213 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:26:29.213 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:26:29.213 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-25T22:26:29.213 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:26:29.213 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2026-02-25T22:26:29.213 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-25T22:26:29.232 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:26:29.232 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:29.233 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:29.233 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:29.233 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:29.234 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:29.234 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:26:29.234 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-25T22:26:29.257 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:26:29.259 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-25T22:26:29.259 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:29.259 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-25T22:26:29.259 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:26:29.260 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:26:29.260 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-25T22:26:29.260 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-25T22:26:29.260 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-25T22:26:29.260 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:29.260 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:29.260 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:29.261 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.c.asok 2026-02-25T22:26:29.261 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:29.261 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.c.asok config get fsid 2026-02-25T22:26:29.316 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-25T22:26:29.316 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:29.316 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-25T22:26:29.316 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:26:29.317 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:26:29.317 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-25T22:26:29.317 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-25T22:26:29.317 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-25T22:26:29.318 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:29.318 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:29.318 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:29.318 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.c.asok 2026-02-25T22:26:29.318 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:29.318 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.c.asok config get mon_host 2026-02-25T22:26:29.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:109: TEST_4_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2026-02-25T22:26:29.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:26:29.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:26:29.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2026-02-25T22:26:29.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:26:29.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/d 2026-02-25T22:26:29.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2026-02-25T22:26:29.393 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:26:29.393 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:26:29.393 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:26:29.394 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:26:29.394 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:29.394 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:29.394 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:26:29.394 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2026-02-25T22:26:29.423 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:26:29.424 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2026-02-25T22:26:29.424 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:29.424 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-25T22:26:29.424 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:26:29.425 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:26:29.425 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-25T22:26:29.425 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-25T22:26:29.426 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-25T22:26:29.426 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:29.426 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:29.426 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:29.426 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.d.asok 2026-02-25T22:26:29.426 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:29.426 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.d.asok config get fsid 2026-02-25T22:26:29.481 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2026-02-25T22:26:29.481 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:26:29.481 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-25T22:26:29.481 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:26:29.483 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:26:29.483 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-25T22:26:29.483 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-25T22:26:29.483 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-25T22:26:29.483 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:26:29.483 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:26:29.483 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:26:29.483 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.d.asok 2026-02-25T22:26:29.483 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:26:29.483 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.d.asok config get mon_host 2026-02-25T22:26:29.538 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:110: TEST_4_mons_checks: wait_for_quorum 60 4 2026-02-25T22:26:29.538 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:26:29.538 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2026-02-25T22:26:29.538 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:26:29.539 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2026-02-25T22:26:29.539 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:26:29.539 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:26:29.539 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772058449 2026-02-25T22:26:29.540 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:29.541 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058389 -lt 1772058449 ]] 2026-02-25T22:26:29.541 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:26:29.541 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:35.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:35.657 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:35.658 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:26:35.658 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:35.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:26:35.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:26:35.668 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:26:35.668 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:26:35.668 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:26:35.668 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:112: TEST_4_mons_checks: ceph mon ok-to-stop dne 2026-02-25T22:26:35.769 INFO:tasks.workunit.client.0.trial188.stderr:invalid mon(s) specified: [dne] 2026-02-25T22:26:35.777 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:113: TEST_4_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:26:35.893 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (b,c,d) after stopping [a] 2026-02-25T22:26:35.901 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:114: TEST_4_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:26:36.015 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (a,c,d) after stopping [b] 2026-02-25T22:26:36.023 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:115: TEST_4_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:26:39.142 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (a,b,d) after stopping [c] 2026-02-25T22:26:39.152 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:116: TEST_4_mons_checks: ceph mon ok-to-stop d 2026-02-25T22:26:39.254 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (a,b,c) after stopping [d] 2026-02-25T22:26:39.261 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:117: TEST_4_mons_checks: ceph mon ok-to-stop a b 2026-02-25T22:26:39.363 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [a,b] 2026-02-25T22:26:39.365 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:118: TEST_4_mons_checks: ceph mon ok-to-stop c d 2026-02-25T22:26:42.453 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (a,b) after stopping mons [c,d] 2026-02-25T22:26:42.457 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:120: TEST_4_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:26:45.561 INFO:tasks.workunit.client.0.trial188.stderr:adding another mon that is not yet online will not break quorum 2026-02-25T22:26:45.569 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:122: TEST_4_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:26:45.682 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.a 2026-02-25T22:26:45.690 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:123: TEST_4_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:26:45.807 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.b 2026-02-25T22:26:45.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:124: TEST_4_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:26:45.932 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.c 2026-02-25T22:26:45.943 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:126: TEST_4_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2026-02-25T22:26:45.943 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:26:45.944 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:26:45.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:26:45.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:26:45.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:26:46.051 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:26:46.051 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:127: TEST_4_mons_checks: wait_for_quorum 60 3 2026-02-25T22:26:46.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:26:46.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-25T22:26:46.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:26:46.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-25T22:26:46.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:26:46.052 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:26:46.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772058466 2026-02-25T22:26:46.053 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:46.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058406 -lt 1772058466 ]] 2026-02-25T22:26:46.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:26:46.054 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:46.172 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:46.172 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:46.173 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:46.173 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:26:46.182 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:46.182 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:46.183 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:46.184 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058406 -lt 1772058466 ]] 2026-02-25T22:26:46.184 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:26:46.184 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:49.298 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:49.298 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:49.299 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:26:49.299 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:49.308 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:49.308 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:49.309 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:49.310 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058409 -lt 1772058466 ]] 2026-02-25T22:26:49.310 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:26:49.310 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:49.426 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:49.426 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:49.427 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:26:49.427 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:49.436 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:49.436 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:49.436 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:49.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058409 -lt 1772058466 ]] 2026-02-25T22:26:49.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:26:49.438 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:49.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:49.549 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:49.550 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:26:49.550 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:49.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:49.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:49.560 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:49.561 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058409 -lt 1772058466 ]] 2026-02-25T22:26:49.561 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:26:49.562 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:52.678 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:52.679 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:52.679 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:26:52.679 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:52.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:52.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:52.689 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:52.690 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058412 -lt 1772058466 ]] 2026-02-25T22:26:52.690 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:26:52.690 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:52.806 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:52.806 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:52.807 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:52.807 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:26:52.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:52.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:52.817 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:52.818 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058412 -lt 1772058466 ]] 2026-02-25T22:26:52.818 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:26:52.818 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:52.930 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:52.930 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:52.931 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:26:52.931 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:52.940 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:52.940 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:52.941 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:52.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058412 -lt 1772058466 ]] 2026-02-25T22:26:52.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:26:52.942 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:26:53.060 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:26:53.060 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:53.061 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:26:53.061 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:26:53.071 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:26:53.071 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:26:53.071 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:26:53.072 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058413 -lt 1772058466 ]] 2026-02-25T22:26:53.072 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:26:53.072 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:05.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:05.192 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":4,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:27:05.193 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":4,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:26:28.894091Z","created":"2026-02-25T22:26:28.894091Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894087+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894088+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894089+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:26:28:894090+0000","crush_location":[]}]}}' 2026-02-25T22:27:05.193 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:27:05.203 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:27:05.203 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:27:05.203 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:27:05.203 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:27:05.203 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:27:05.203 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:129: TEST_4_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:27:05.308 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (b,c,d) after stopping [a] 2026-02-25T22:27:05.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:130: TEST_4_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:27:05.417 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2026-02-25T22:27:05.420 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:131: TEST_4_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:27:05.508 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2026-02-25T22:27:05.512 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:132: TEST_4_mons_checks: ceph mon ok-to-stop d 2026-02-25T22:27:05.599 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2026-02-25T22:27:05.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:134: TEST_4_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:27:05.709 INFO:tasks.workunit.client.0.trial188.stderr:adding another mon that is not yet online will not break quorum 2026-02-25T22:27:05.720 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:136: TEST_4_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:27:05.827 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.a 2026-02-25T22:27:05.836 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:137: TEST_4_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:27:05.956 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.b 2026-02-25T22:27:05.964 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:138: TEST_4_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:27:06.080 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.c 2026-02-25T22:27:06.088 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:139: TEST_4_mons_checks: ceph mon ok-to-rm d 2026-02-25T22:27:09.225 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.d 2026-02-25T22:27:09.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:27:09.237 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:27:09.237 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:27:09.237 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:27:09.237 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:27:09.238 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:27:09.346 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:27:09.346 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:27:09.346 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:27:09.346 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:27:09.346 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:27:09.346 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:27:09.346 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:27:09.346 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:27:09.346 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:27:09.346 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:27:09.351 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:27:09.352 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:27:09.353 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:27:09.353 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:27:09.354 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:27:09.355 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:27:09.355 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:27:09.356 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:27:09.356 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:27:09.357 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:27:09.357 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:27:09.358 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:27:09.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:27:09.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:27:09.368 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:27:09.368 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.368 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.369 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:27:09.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:27:09.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:27:09.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:19: run: for func in $funcs 2026-02-25T22:27:09.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:20: run: setup td/ok-to-stop 2026-02-25T22:27:09.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ok-to-stop 2026-02-25T22:27:09.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ok-to-stop 2026-02-25T22:27:09.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:27:09.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:27:09.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:27:09.372 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:27:09.372 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:27:09.372 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:27:09.372 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:27:09.372 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:27:09.372 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:27:09.373 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:27:09.374 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:27:09.374 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:27:09.376 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:27:09.376 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:27:09.376 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:27:09.377 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:27:09.377 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:27:09.378 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:27:09.378 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:27:09.379 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:27:09.380 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:27:09.380 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:27:09.381 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:27:09.381 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.381 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.381 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:27:09.382 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:27:09.382 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:27:09.383 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ok-to-stop 2026-02-25T22:27:09.384 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:27:09.384 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.384 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.384 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.35988 2026-02-25T22:27:09.386 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:27:09.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:27:09.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:27:09.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ok-to-stop 1' TERM HUP INT 2026-02-25T22:27:09.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:21: run: TEST_5_mons_checks td/ok-to-stop 2026-02-25T22:27:09.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:143: TEST_5_mons_checks: local dir=td/ok-to-stop 2026-02-25T22:27:09.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:145: TEST_5_mons_checks: CEPH_ARGS='--fsid=39af357d-9d74-4b36-bb64-741126d525fb --auth-supported=none --mon-host=127.0.0.1:7150,127.0.0.1:7151,127.0.0.1:7152,127.0.0.1:7153,127.0.0.1:7154 ' 2026-02-25T22:27:09.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:147: TEST_5_mons_checks: run_mon td/ok-to-stop a --public-addr=127.0.0.1:7150 2026-02-25T22:27:09.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:27:09.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:27:09.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:27:09.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:27:09.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/a 2026-02-25T22:27:09.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ok-to-stop/a --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7150 2026-02-25T22:27:09.408 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:27:09.409 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:09.409 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:09.409 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:09.409 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.409 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.409 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:27:09.410 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/a '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7150 2026-02-25T22:27:09.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:27:09.439 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:27:09.440 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:27:09.440 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:27:09.440 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:27:09.440 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:27:09.441 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:27:09.441 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:27:09.441 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:27:09.441 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:09.441 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.442 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.442 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:27:09.442 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:27:09.442 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get fsid 2026-02-25T22:27:09.495 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:27:09.496 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:27:09.496 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:27:09.496 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:27:09.497 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:27:09.497 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:27:09.497 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:27:09.497 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:27:09.497 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:09.498 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.498 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.498 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.a.asok 2026-02-25T22:27:09.498 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:27:09.498 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.a.asok config get mon_host 2026-02-25T22:27:09.551 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:148: TEST_5_mons_checks: run_mon td/ok-to-stop b --public-addr=127.0.0.1:7151 2026-02-25T22:27:09.551 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:27:09.551 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:27:09.551 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-25T22:27:09.551 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:27:09.551 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/b 2026-02-25T22:27:09.551 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/ok-to-stop/b --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7151 2026-02-25T22:27:09.570 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:27:09.570 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:09.570 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:09.570 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:09.570 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.570 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.571 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:27:09.571 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/b '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7151 2026-02-25T22:27:09.598 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:27:09.599 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-25T22:27:09.599 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:27:09.599 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:27:09.599 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:27:09.600 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:27:09.601 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:27:09.601 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:27:09.601 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:27:09.601 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:09.601 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.601 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.601 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.b.asok 2026-02-25T22:27:09.601 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:27:09.601 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.b.asok config get fsid 2026-02-25T22:27:09.656 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-25T22:27:09.656 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:27:09.656 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-25T22:27:09.656 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:27:09.657 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:27:09.658 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-25T22:27:09.658 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-25T22:27:09.658 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-25T22:27:09.658 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:09.658 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.658 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.658 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.b.asok 2026-02-25T22:27:09.658 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:27:09.658 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.b.asok config get mon_host 2026-02-25T22:27:09.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:149: TEST_5_mons_checks: run_mon td/ok-to-stop c --public-addr=127.0.0.1:7152 2026-02-25T22:27:09.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:27:09.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:27:09.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-25T22:27:09.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:27:09.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/c 2026-02-25T22:27:09.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/ok-to-stop/c --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7152 2026-02-25T22:27:09.730 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:27:09.730 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:09.730 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:09.730 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:09.730 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.730 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.731 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:27:09.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/c '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7152 2026-02-25T22:27:09.760 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:27:09.761 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-25T22:27:09.761 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:27:09.761 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-25T22:27:09.761 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:27:09.761 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:27:09.762 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-25T22:27:09.763 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-25T22:27:09.763 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-25T22:27:09.763 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:09.763 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.763 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.763 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.c.asok 2026-02-25T22:27:09.763 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:27:09.763 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.c.asok config get fsid 2026-02-25T22:27:09.828 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-25T22:27:09.828 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:27:09.828 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-25T22:27:09.828 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:27:09.830 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:27:09.830 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-25T22:27:09.830 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-25T22:27:09.830 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-25T22:27:09.830 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:09.830 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.830 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.830 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.c.asok 2026-02-25T22:27:09.830 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:27:09.830 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.c.asok config get mon_host 2026-02-25T22:27:09.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:150: TEST_5_mons_checks: run_mon td/ok-to-stop d --public-addr=127.0.0.1:7153 2026-02-25T22:27:09.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:27:09.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:27:09.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2026-02-25T22:27:09.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:27:09.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/d 2026-02-25T22:27:09.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/ok-to-stop/d --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7153 2026-02-25T22:27:09.904 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:27:09.904 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:09.904 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:09.904 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:09.904 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.905 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.905 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:27:09.905 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/d '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7153 2026-02-25T22:27:09.940 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:27:09.941 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2026-02-25T22:27:09.941 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:27:09.941 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-25T22:27:09.941 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:27:09.941 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:27:09.942 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-25T22:27:09.942 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-25T22:27:09.942 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-25T22:27:09.942 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:09.943 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.943 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.943 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.d.asok 2026-02-25T22:27:09.943 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:27:09.943 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.d.asok config get fsid 2026-02-25T22:27:09.997 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2026-02-25T22:27:09.997 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:27:09.997 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-25T22:27:09.998 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:27:09.999 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:27:09.999 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-25T22:27:09.999 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-25T22:27:09.999 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-25T22:27:09.999 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:09.999 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:09.999 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:09.999 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.d.asok 2026-02-25T22:27:09.999 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:27:10.000 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.d.asok config get mon_host 2026-02-25T22:27:10.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:151: TEST_5_mons_checks: run_mon td/ok-to-stop e --public-addr=127.0.0.1:7154 2026-02-25T22:27:10.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ok-to-stop 2026-02-25T22:27:10.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:27:10.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=e 2026-02-25T22:27:10.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:27:10.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ok-to-stop/e 2026-02-25T22:27:10.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id e --mkfs --mon-data=td/ok-to-stop/e --run-dir=td/ok-to-stop --public-addr=127.0.0.1:7154 2026-02-25T22:27:10.073 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:27:10.073 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:27:10.074 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:27:10.074 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:27:10.074 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:10.074 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:10.074 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.35988/$cluster-$name.asok' 2026-02-25T22:27:10.074 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ok-to-stop/e '--log-file=td/ok-to-stop/$name.log' '--admin-socket=/tmp/ceph-asok.35988/$cluster-$name.asok' --mon-cluster-log-file=td/ok-to-stop/log --run-dir=td/ok-to-stop '--pid-file=td/ok-to-stop/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr=127.0.0.1:7154 2026-02-25T22:27:10.103 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:27:10.104 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e fsid 2026-02-25T22:27:10.104 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:27:10.104 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2026-02-25T22:27:10.104 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:27:10.105 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:27:10.105 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2026-02-25T22:27:10.105 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2026-02-25T22:27:10.105 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2026-02-25T22:27:10.106 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:10.106 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:10.106 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:10.106 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.e.asok 2026-02-25T22:27:10.106 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:27:10.106 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.e.asok config get fsid 2026-02-25T22:27:10.161 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e mon_host 2026-02-25T22:27:10.162 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:27:10.162 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2026-02-25T22:27:10.162 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:27:10.162 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:27:10.162 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2026-02-25T22:27:10.162 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2026-02-25T22:27:10.162 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2026-02-25T22:27:10.163 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:27:10.163 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:27:10.163 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:27:10.163 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.35988/ceph-mon.e.asok 2026-02-25T22:27:10.164 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:27:10.164 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.35988/ceph-mon.e.asok config get mon_host 2026-02-25T22:27:10.219 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:152: TEST_5_mons_checks: wait_for_quorum 60 5 2026-02-25T22:27:10.219 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:27:10.219 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=5 2026-02-25T22:27:10.220 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:27:10.220 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 5 ]] 2026-02-25T22:27:10.220 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:27:10.220 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:27:10.220 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772058490 2026-02-25T22:27:10.221 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:10.222 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058430 -lt 1772058490 ]] 2026-02-25T22:27:10.222 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 5' 2026-02-25T22:27:10.222 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:19.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:19.341 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:19.342 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:19.342 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 5' 2026-02-25T22:27:19.351 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:27:19.352 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:27:19.352 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:27:19.352 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:27:19.352 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:27:19.352 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:154: TEST_5_mons_checks: ceph mon ok-to-stop dne 2026-02-25T22:27:19.458 INFO:tasks.workunit.client.0.trial188.stderr:invalid mon(s) specified: [dne] 2026-02-25T22:27:19.466 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:155: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:27:19.571 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2026-02-25T22:27:19.579 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:156: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:27:19.697 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (a,c,d,e) after stopping [b] 2026-02-25T22:27:19.708 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:157: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:27:19.813 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (a,b,d,e) after stopping [c] 2026-02-25T22:27:19.824 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:158: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-25T22:27:19.929 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (a,b,c,e) after stopping [d] 2026-02-25T22:27:19.939 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:159: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-25T22:27:20.043 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (a,b,c,d) after stopping [e] 2026-02-25T22:27:20.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:160: TEST_5_mons_checks: ceph mon ok-to-stop a b 2026-02-25T22:27:20.169 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (c,d,e) after stopping [a,b] 2026-02-25T22:27:20.179 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:161: TEST_5_mons_checks: ceph mon ok-to-stop c d 2026-02-25T22:27:20.284 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (a,b,e) after stopping [c,d] 2026-02-25T22:27:20.292 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:162: TEST_5_mons_checks: ceph mon ok-to-stop a b c 2026-02-25T22:27:20.392 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (d,e) after stopping mons [a,b,c] 2026-02-25T22:27:20.396 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:164: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:27:20.499 INFO:tasks.workunit.client.0.trial188.stderr:adding another mon that is not yet online will not break quorum 2026-02-25T22:27:20.509 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:166: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:27:20.616 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.a 2026-02-25T22:27:20.624 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:167: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:27:20.742 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.b 2026-02-25T22:27:20.750 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:168: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:27:20.869 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.c 2026-02-25T22:27:20.877 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:169: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-25T22:27:20.997 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.d 2026-02-25T22:27:21.005 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:170: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-25T22:27:21.111 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.e 2026-02-25T22:27:21.119 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:172: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.a 2026-02-25T22:27:21.120 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:27:21.120 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:27:21.120 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:27:21.120 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:27:21.120 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:27:21.227 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:27:21.227 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:173: TEST_5_mons_checks: wait_for_quorum 60 4 2026-02-25T22:27:21.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:27:21.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2026-02-25T22:27:21.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:27:21.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2026-02-25T22:27:21.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:27:21.228 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:27:21.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772058501 2026-02-25T22:27:21.229 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:21.230 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058441 -lt 1772058501 ]] 2026-02-25T22:27:21.230 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:27:21.230 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:24.366 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:24.366 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:24.367 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:27:24.367 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:24.376 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:27:24.376 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:27:24.376 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:24.377 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058444 -lt 1772058501 ]] 2026-02-25T22:27:24.377 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:27:24.378 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:24.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:24.493 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:24.493 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:27:24.493 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:24.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:27:24.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:27:24.503 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:24.505 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058444 -lt 1772058501 ]] 2026-02-25T22:27:24.505 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:27:24.505 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:24.618 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:24.618 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:24.619 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:27:24.619 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:24.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:27:24.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:27:24.629 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:24.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058444 -lt 1772058501 ]] 2026-02-25T22:27:24.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:27:24.631 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:24.746 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:24.747 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:24.747 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:24.747 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:27:24.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:27:24.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:27:24.756 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:24.758 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058444 -lt 1772058501 ]] 2026-02-25T22:27:24.758 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:27:24.758 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:27.873 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:27.873 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:27.874 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:27:27.874 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:27.883 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:27:27.883 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:27:27.884 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:27.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058447 -lt 1772058501 ]] 2026-02-25T22:27:27.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:27:27.885 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:27.998 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:27.998 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:27.998 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:27:27.999 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:28.007 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:27:28.007 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:27:28.008 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:28.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058448 -lt 1772058501 ]] 2026-02-25T22:27:28.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:27:28.009 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:28.126 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:28.126 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:28.127 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:28.127 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:27:28.136 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:27:28.136 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:27:28.136 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:28.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058448 -lt 1772058501 ]] 2026-02-25T22:27:28.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-25T22:27:28.138 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:40.250 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:40.251 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":4,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:40.251 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-25T22:27:40.251 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":4,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:40.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:27:40.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:27:40.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:27:40.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:27:40.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:27:40.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:175: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:27:40.366 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (b,c,d,e) after stopping [a] 2026-02-25T22:27:40.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:176: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:27:40.491 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (c,d,e) after stopping [b] 2026-02-25T22:27:40.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:177: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:27:40.603 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (b,d,e) after stopping [c] 2026-02-25T22:27:40.611 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:178: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-25T22:27:40.719 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (b,c,e) after stopping [d] 2026-02-25T22:27:40.729 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:179: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-25T22:27:40.832 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (b,c,d) after stopping [e] 2026-02-25T22:27:40.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:181: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:27:40.957 INFO:tasks.workunit.client.0.trial188.stderr:adding another mon that is not yet online will not break quorum 2026-02-25T22:27:40.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:183: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:27:41.080 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.a 2026-02-25T22:27:41.087 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:184: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:27:41.204 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.b 2026-02-25T22:27:41.215 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:185: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:27:41.319 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.c 2026-02-25T22:27:41.330 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:186: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-25T22:27:44.435 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.d 2026-02-25T22:27:44.442 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:187: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-25T22:27:44.556 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.e 2026-02-25T22:27:44.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:189: TEST_5_mons_checks: kill_daemons td/ok-to-stop KILL mon.e 2026-02-25T22:27:44.567 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:27:44.567 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:27:44.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:27:44.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:27:44.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:27:44.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:27:44.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:190: TEST_5_mons_checks: wait_for_quorum 60 3 2026-02-25T22:27:44.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=60 2026-02-25T22:27:44.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-25T22:27:44.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 60 ]] 2026-02-25T22:27:44.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-25T22:27:44.676 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-25T22:27:44.676 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-25T22:27:44.676 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1772058524 2026-02-25T22:27:44.676 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:44.677 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058464 -lt 1772058524 ]] 2026-02-25T22:27:44.677 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:27:44.678 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:44.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:44.786 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:44.787 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:27:44.787 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":8,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:44.796 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:27:44.796 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:27:44.797 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:44.798 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058464 -lt 1772058524 ]] 2026-02-25T22:27:44.798 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:27:44.798 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:47.916 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:47.916 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:47.917 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:47.917 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:27:47.927 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:27:47.927 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:27:47.927 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:47.928 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058467 -lt 1772058524 ]] 2026-02-25T22:27:47.928 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:27:47.929 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:48.044 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:48.044 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:48.045 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":11,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:48.045 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:27:48.055 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:27:48.055 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:27:48.055 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:48.056 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058468 -lt 1772058524 ]] 2026-02-25T22:27:48.057 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:27:48.057 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:27:54.172 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:27:54.172 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":18,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:54.173 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":10,"quorum":[1,2,3,4],"quorum_names":["b","c","d","e"],"quorum_leader_name":"b","quorum_age":18,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:27:54.173 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:27:54.183 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=false 2026-02-25T22:27:54.183 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ false == \t\r\u\e ]] 2026-02-25T22:27:54.183 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-25T22:27:54.184 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1772058474 -lt 1772058524 ]] 2026-02-25T22:27:54.184 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-25T22:27:54.184 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 60 ceph quorum_status --format=json 2026-02-25T22:28:00.301 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-25T22:28:00.301 INFO:tasks.workunit.client.0.trial188.stderr:{"election_epoch":14,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:28:00.302 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":14,"quorum":[1,2,3],"quorum_names":["b","c","d"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"39af357d-9d74-4b36-bb64-741126d525fb","modified":"2026-02-25T22:27:09.398330Z","created":"2026-02-25T22:27:09.398330Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7150","nonce":0}]},"addr":"127.0.0.1:7150/0","public_addr":"127.0.0.1:7150/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398325+0000","crush_location":[]},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7151","nonce":0}]},"addr":"127.0.0.1:7151/0","public_addr":"127.0.0.1:7151/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398327+0000","crush_location":[]},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7152","nonce":0}]},"addr":"127.0.0.1:7152/0","public_addr":"127.0.0.1:7152/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398328+0000","crush_location":[]},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7153","nonce":0}]},"addr":"127.0.0.1:7153/0","public_addr":"127.0.0.1:7153/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7154","nonce":0}]},"addr":"127.0.0.1:7154/0","public_addr":"127.0.0.1:7154/0","priority":0,"weight":0,"time_added":"2026-02-25T22:27:09:398329+0000","crush_location":[]}]}}' 2026-02-25T22:28:00.302 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-25T22:28:00.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-25T22:28:00.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-25T22:28:00.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-25T22:28:00.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-25T22:28:00.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-25T22:28:00.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:192: TEST_5_mons_checks: ceph mon ok-to-stop a 2026-02-25T22:28:00.419 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (b,c,d) after stopping [a] 2026-02-25T22:28:00.427 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:193: TEST_5_mons_checks: ceph mon ok-to-stop b 2026-02-25T22:28:03.533 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (c,d) after stopping mons [b] 2026-02-25T22:28:03.536 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:194: TEST_5_mons_checks: ceph mon ok-to-stop c 2026-02-25T22:28:06.641 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (b,d) after stopping mons [c] 2026-02-25T22:28:06.643 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:195: TEST_5_mons_checks: ceph mon ok-to-stop d 2026-02-25T22:28:06.727 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: not enough monitors would be available (b,c) after stopping mons [d] 2026-02-25T22:28:06.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:196: TEST_5_mons_checks: ceph mon ok-to-stop e 2026-02-25T22:28:06.831 INFO:tasks.workunit.client.0.trial188.stderr:quorum should be preserved (b,c,d) after stopping [e] 2026-02-25T22:28:06.841 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:198: TEST_5_mons_checks: ceph mon ok-to-add-offline 2026-02-25T22:28:06.927 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: adding a monitor may break quorum (until that monitor starts) 2026-02-25T22:28:06.929 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:200: TEST_5_mons_checks: ceph mon ok-to-rm a 2026-02-25T22:28:07.032 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.a 2026-02-25T22:28:07.043 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:201: TEST_5_mons_checks: ceph mon ok-to-rm b 2026-02-25T22:28:07.128 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: removing mon.b would break quorum 2026-02-25T22:28:07.132 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:202: TEST_5_mons_checks: ceph mon ok-to-rm c 2026-02-25T22:28:10.219 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: removing mon.c would break quorum 2026-02-25T22:28:10.221 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:203: TEST_5_mons_checks: ceph mon ok-to-rm d 2026-02-25T22:28:10.304 INFO:tasks.workunit.client.0.trial188.stderr:Error EBUSY: removing mon.d would break quorum 2026-02-25T22:28:10.307 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:204: TEST_5_mons_checks: ceph mon ok-to-rm e 2026-02-25T22:28:10.410 INFO:tasks.workunit.client.0.trial188.stderr:safe to remove mon.e 2026-02-25T22:28:10.420 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:22: run: kill_daemons td/ok-to-stop KILL 2026-02-25T22:28:10.421 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:10.421 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:10.421 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:10.422 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:10.422 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:10.530 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:10.531 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/ok-to-stop.sh:23: run: teardown td/ok-to-stop 2026-02-25T22:28:10.531 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:28:10.531 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:10.531 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:28:10.531 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:10.531 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:10.532 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:10.532 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:10.532 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:10.538 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:10.538 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:10.539 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:10.540 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:10.541 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:10.541 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:10.541 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:10.542 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:10.542 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:10.543 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:10.543 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:10.544 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:10.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:10.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:28:10.559 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:10.559 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:10.559 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:28:10.559 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:28:10.561 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:10.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:10.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T22:28:10.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/ok-to-stop 0 2026-02-25T22:28:10.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ok-to-stop 2026-02-25T22:28:10.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T22:28:10.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ok-to-stop KILL 2026-02-25T22:28:10.562 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:10.562 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:10.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:10.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:10.562 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:10.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:10.564 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:10.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:10.565 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:10.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:10.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:10.567 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:10.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:10.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:10.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:10.568 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:10.569 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:10.570 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T22:28:10.571 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ok-to-stop 2026-02-25T22:28:10.572 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:10.572 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:10.572 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.35988 2026-02-25T22:28:10.572 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.35988 2026-02-25T22:28:10.573 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:10.574 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:10.574 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T22:28:10.575 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T22:28:10.575 DEBUG:teuthology.orchestra.run.trial188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T22:28:10.624 INFO:tasks.workunit:Running workunit misc/rados-striper.sh... 2026-02-25T22:28:10.624 DEBUG:teuthology.orchestra.run.trial188:workunit test misc/rados-striper.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61d4431d3ec879bd98659388b3655acf3d392418 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh 2026-02-25T22:28:10.676 INFO:tasks.workunit.client.0.trial188.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T22:28:10.681 INFO:tasks.workunit.client.0.trial188.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T22:28:10.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:28:10.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2393: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:28:10.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PYTHONWARNINGS=ignore 2026-02-25T22:28:10.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PYTHONWARNINGS=ignore 2026-02-25T22:28:10.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export CEPH_CONF=/dev/null 2026-02-25T22:28:10.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: CEPH_CONF=/dev/null 2026-02-25T22:28:10.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: unset CEPH_ARGS 2026-02-25T22:28:10.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2398: main: local code 2026-02-25T22:28:10.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: run td/rados-striper 2026-02-25T22:28:10.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:20: run: local dir=td/rados-striper 2026-02-25T22:28:10.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:21: run: shift 2026-02-25T22:28:10.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: export CEPH_MON=127.0.0.1:7116 2026-02-25T22:28:10.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:23: run: CEPH_MON=127.0.0.1:7116 2026-02-25T22:28:10.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:24: run: export CEPH_ARGS 2026-02-25T22:28:10.682 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: uuidgen 2026-02-25T22:28:10.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:25: run: CEPH_ARGS+='--fsid=d25c3dae-10f8-4263-bb77-504e6ce07da4 --auth-supported=none ' 2026-02-25T22:28:10.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:26: run: CEPH_ARGS+='--mon-host=127.0.0.1:7116 ' 2026-02-25T22:28:10.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:29: run: setup td/rados-striper 2026-02-25T22:28:10.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/rados-striper 2026-02-25T22:28:10.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/rados-striper 2026-02-25T22:28:10.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2026-02-25T22:28:10.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:10.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2026-02-25T22:28:10.685 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:10.685 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:10.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:10.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:10.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:10.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:10.686 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:10.686 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:10.686 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:10.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:10.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:10.688 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:10.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:10.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:10.690 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:10.690 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:10.691 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:10.692 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:10.692 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2026-02-25T22:28:10.694 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:10.694 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:10.694 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50563 2026-02-25T22:28:10.694 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.50563 2026-02-25T22:28:10.695 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:10.695 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:10.695 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/rados-striper 2026-02-25T22:28:10.697 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:28:10.697 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:10.697 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50563 2026-02-25T22:28:10.697 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.50563 2026-02-25T22:28:10.698 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:28:10.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T22:28:10.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T22:28:10.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:28:10.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/rados-striper 1' TERM HUP INT 2026-02-25T22:28:10.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:32: run: run_mon td/rados-striper a 2026-02-25T22:28:10.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/rados-striper 2026-02-25T22:28:10.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:28:10.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:28:10.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:28:10.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/rados-striper/a 2026-02-25T22:28:10.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/rados-striper/a --run-dir=td/rados-striper 2026-02-25T22:28:10.722 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:28:10.722 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:10.723 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:10.723 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:10.723 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:10.723 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50563 2026-02-25T22:28:10.723 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50563/$cluster-$name.asok' 2026-02-25T22:28:10.723 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/rados-striper/a '--log-file=td/rados-striper/$name.log' '--admin-socket=/tmp/ceph-asok.50563/$cluster-$name.asok' --mon-cluster-log-file=td/rados-striper/log --run-dir=td/rados-striper '--pid-file=td/rados-striper/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:28:10.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:28:10.752 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:28:10.752 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:10.752 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:10.752 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:28:10.752 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:28:10.753 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:10.754 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:10.754 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:10.754 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:10.754 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:10.754 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50563 2026-02-25T22:28:10.754 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.50563/ceph-mon.a.asok 2026-02-25T22:28:10.754 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:10.754 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.50563/ceph-mon.a.asok config get fsid 2026-02-25T22:28:10.808 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:28:10.808 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:10.808 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:10.808 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:28:10.808 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:28:10.809 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:10.810 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:10.810 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:10.810 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:10.810 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:10.810 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50563 2026-02-25T22:28:10.810 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.50563/ceph-mon.a.asok 2026-02-25T22:28:10.810 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:10.810 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.50563/ceph-mon.a.asok config get mon_host 2026-02-25T22:28:10.864 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:33: run: run_osd td/rados-striper 0 2026-02-25T22:28:10.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2026-02-25T22:28:10.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:10.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:28:10.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:10.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/0 2026-02-25T22:28:10.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d25c3dae-10f8-4263-bb77-504e6ce07da4 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-25T22:28:10.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:10.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:10.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:10.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/0' 2026-02-25T22:28:10.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/0/journal' 2026-02-25T22:28:10.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:10.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:10.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-25T22:28:10.866 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:10.866 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:10.866 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:10.866 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:10.866 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:10.866 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50563 2026-02-25T22:28:10.866 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50563/$cluster-$name.asok' 2026-02-25T22:28:10.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50563/$cluster-$name.asok' 2026-02-25T22:28:10.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:10.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:10.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:10.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-25T22:28:10.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-25T22:28:10.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:10.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:10.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:10.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:10.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:10.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:10.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/0 2026-02-25T22:28:10.868 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:10.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=eee3c6ee-10d4-42da-96b6-86775fccbc0e 2026-02-25T22:28:10.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 eee3c6ee-10d4-42da-96b6-86775fccbc0e' 2026-02-25T22:28:10.868 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 eee3c6ee-10d4-42da-96b6-86775fccbc0e 2026-02-25T22:28:10.869 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:10.878 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB6d59pVngqNBAANk7vXXUJFcNtZUNFj2dhWw== 2026-02-25T22:28:10.878 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB6d59pVngqNBAANk7vXXUJFcNtZUNFj2dhWw=="}' 2026-02-25T22:28:10.878 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new eee3c6ee-10d4-42da-96b6-86775fccbc0e -i td/rados-striper/0/new.json 2026-02-25T22:28:10.994 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:11.005 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/0/new.json 2026-02-25T22:28:11.006 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=d25c3dae-10f8-4263-bb77-504e6ce07da4 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.50563/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB6d59pVngqNBAANk7vXXUJFcNtZUNFj2dhWw== --osd-uuid eee3c6ee-10d4-42da-96b6-86775fccbc0e 2026-02-25T22:28:11.029 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:11.022+0000 7f53e5181a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:11.031 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:11.022+0000 7f53e5181a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:11.032 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:11.026+0000 7f53e5181a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:11.033 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:11.026+0000 7f53e5181a40 -1 bdev(0x55b5aaf78400 td/rados-striper/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:11.033 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:11.026+0000 7f53e5181a40 -1 bluestore(td/rados-striper/0) _read_fsid unparsable uuid 2026-02-25T22:28:11.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/0/keyring 2026-02-25T22:28:11.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:11.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:28:11.641 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:28:11.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:11.761 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:28:11.762 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:28:11.762 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=d25c3dae-10f8-4263-bb77-504e6ce07da4 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/0 --osd-journal=td/rados-striper/0/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.50563/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:11.762 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:11.762 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:11.762 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:11.782 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:11.774+0000 7f8aeff68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:11.784 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:11.778+0000 7f8aeff68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:11.785 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:11.778+0000 7f8aeff68a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:11.873 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:28:11.873 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:11.873 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:11.874 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:11.874 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:11.874 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:11.874 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:11.875 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:11.875 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:11.875 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:11.980 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:12.173 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:12.166+0000 7f8aeff68a40 -1 Falling back to public interface 2026-02-25T22:28:12.771 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:12.762+0000 7f8aeff68a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:28:12.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:12.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:12.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:12.982 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:28:12.982 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:12.982 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:13.090 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:14.092 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:14.092 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:14.092 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:14.092 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:28:14.093 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:14.093 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:14.200 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1571099296,v1:127.0.0.1:6801/1571099296] [v2:127.0.0.1:6802/1571099296,v1:127.0.0.1:6803/1571099296] exists,up eee3c6ee-10d4-42da-96b6-86775fccbc0e 2026-02-25T22:28:14.203 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:14.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:14.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:14.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:34: run: run_osd td/rados-striper 1 2026-02-25T22:28:14.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2026-02-25T22:28:14.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:14.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:28:14.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:14.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/1 2026-02-25T22:28:14.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d25c3dae-10f8-4263-bb77-504e6ce07da4 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-25T22:28:14.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:14.204 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:14.205 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:14.205 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/1' 2026-02-25T22:28:14.205 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/1/journal' 2026-02-25T22:28:14.205 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:14.205 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:14.205 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-25T22:28:14.205 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:14.205 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:14.205 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:14.205 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:14.205 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:14.205 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50563 2026-02-25T22:28:14.206 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50563/$cluster-$name.asok' 2026-02-25T22:28:14.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50563/$cluster-$name.asok' 2026-02-25T22:28:14.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:14.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:14.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:14.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-25T22:28:14.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-25T22:28:14.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:14.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:14.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:14.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:14.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:14.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:14.207 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/1 2026-02-25T22:28:14.207 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:14.207 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=6c48deaa-9932-44f6-bad3-d6cc89a8eeab 2026-02-25T22:28:14.207 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 6c48deaa-9932-44f6-bad3-d6cc89a8eeab' 2026-02-25T22:28:14.208 INFO:tasks.workunit.client.0.trial188.stdout:add osd1 6c48deaa-9932-44f6-bad3-d6cc89a8eeab 2026-02-25T22:28:14.208 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:14.217 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB+d59pAgfCDBAAD2+hFBfDD9y1reefQB6M8g== 2026-02-25T22:28:14.217 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB+d59pAgfCDBAAD2+hFBfDD9y1reefQB6M8g=="}' 2026-02-25T22:28:14.217 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 6c48deaa-9932-44f6-bad3-d6cc89a8eeab -i td/rados-striper/1/new.json 2026-02-25T22:28:14.331 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:28:14.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/1/new.json 2026-02-25T22:28:14.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=d25c3dae-10f8-4263-bb77-504e6ce07da4 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.50563/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB+d59pAgfCDBAAD2+hFBfDD9y1reefQB6M8g== --osd-uuid 6c48deaa-9932-44f6-bad3-d6cc89a8eeab 2026-02-25T22:28:14.364 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:14.358+0000 7f337adb5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:14.365 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:14.358+0000 7f337adb5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:14.367 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:14.358+0000 7f337adb5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:14.367 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:14.358+0000 7f337adb5a40 -1 bdev(0x55fddf94a400 td/rados-striper/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:14.367 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:14.362+0000 7f337adb5a40 -1 bluestore(td/rados-striper/1) _read_fsid unparsable uuid 2026-02-25T22:28:15.000 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/1/keyring 2026-02-25T22:28:15.000 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:15.001 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:28:15.001 INFO:tasks.workunit.client.0.trial188.stdout:adding osd1 key to auth repository 2026-02-25T22:28:15.001 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:15.119 INFO:tasks.workunit.client.0.trial188.stdout:start osd.1 2026-02-25T22:28:15.120 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:28:15.120 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=d25c3dae-10f8-4263-bb77-504e6ce07da4 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/1 --osd-journal=td/rados-striper/1/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.50563/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:15.120 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:15.120 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:15.120 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:15.141 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:15.134+0000 7f67d53e0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:15.143 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:15.138+0000 7f67d53e0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:15.145 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:15.138+0000 7f67d53e0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:15.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:28:15.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:15.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:28:15.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:15.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:15.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:15.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:15.229 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:15.230 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:15.230 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:28:15.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:15.497 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:15.490+0000 7f67d53e0a40 -1 Falling back to public interface 2026-02-25T22:28:16.108 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:16.102+0000 7f67d53e0a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:28:16.343 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:16.343 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:16.343 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:16.343 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:28:16.344 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:16.344 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:28:16.452 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:17.454 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:17.454 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:17.454 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:28:17.455 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:17.455 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:17.455 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:28:17.561 INFO:tasks.workunit.client.0.trial188.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6808/4271536071,v1:127.0.0.1:6809/4271536071] [v2:127.0.0.1:6810/4271536071,v1:127.0.0.1:6811/4271536071] exists,up 6c48deaa-9932-44f6-bad3-d6cc89a8eeab 2026-02-25T22:28:17.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:17.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:17.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:17.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:35: run: run_osd td/rados-striper 2 2026-02-25T22:28:17.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/rados-striper 2026-02-25T22:28:17.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:17.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-25T22:28:17.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:17.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/rados-striper/2 2026-02-25T22:28:17.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=d25c3dae-10f8-4263-bb77-504e6ce07da4 --auth-supported=none --mon-host=127.0.0.1:7116 ' 2026-02-25T22:28:17.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:17.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:17.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:17.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/rados-striper/2' 2026-02-25T22:28:17.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/rados-striper/2/journal' 2026-02-25T22:28:17.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:17.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:17.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/rados-striper' 2026-02-25T22:28:17.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:17.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:17.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:17.566 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:17.566 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:17.566 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50563 2026-02-25T22:28:17.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.50563/$cluster-$name.asok' 2026-02-25T22:28:17.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.50563/$cluster-$name.asok' 2026-02-25T22:28:17.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:17.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:17.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:17.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/rados-striper/$name.log' 2026-02-25T22:28:17.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/rados-striper/$name.pid' 2026-02-25T22:28:17.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:17.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:17.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:17.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:17.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:17.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:17.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/rados-striper/2 2026-02-25T22:28:17.568 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:17.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1f70bffb-a264-4e19-9b82-833a503939cc 2026-02-25T22:28:17.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 1f70bffb-a264-4e19-9b82-833a503939cc' 2026-02-25T22:28:17.569 INFO:tasks.workunit.client.0.trial188.stdout:add osd2 1f70bffb-a264-4e19-9b82-833a503939cc 2026-02-25T22:28:17.569 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:17.577 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCBd59pkQ1CIhAA52PUIBaLhMQm9L5vtKbQug== 2026-02-25T22:28:17.577 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCBd59pkQ1CIhAA52PUIBaLhMQm9L5vtKbQug=="}' 2026-02-25T22:28:17.577 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1f70bffb-a264-4e19-9b82-833a503939cc -i td/rados-striper/2/new.json 2026-02-25T22:28:17.692 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:28:17.699 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/rados-striper/2/new.json 2026-02-25T22:28:17.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=d25c3dae-10f8-4263-bb77-504e6ce07da4 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.50563/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCBd59pkQ1CIhAA52PUIBaLhMQm9L5vtKbQug== --osd-uuid 1f70bffb-a264-4e19-9b82-833a503939cc 2026-02-25T22:28:17.716 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:17.710+0000 7fe345f53a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:17.718 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:17.710+0000 7fe345f53a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:17.719 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:17.710+0000 7fe345f53a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:17.719 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:17.710+0000 7fe345f53a40 -1 bdev(0x5654acbee400 td/rados-striper/2/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:17.719 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:17.714+0000 7fe345f53a40 -1 bluestore(td/rados-striper/2) _read_fsid unparsable uuid 2026-02-25T22:28:18.318 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/rados-striper/2/keyring 2026-02-25T22:28:18.318 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:18.319 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-25T22:28:18.320 INFO:tasks.workunit.client.0.trial188.stdout:adding osd2 key to auth repository 2026-02-25T22:28:18.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/rados-striper/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:18.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-25T22:28:18.437 INFO:tasks.workunit.client.0.trial188.stdout:start osd.2 2026-02-25T22:28:18.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=d25c3dae-10f8-4263-bb77-504e6ce07da4 --auth-supported=none --mon-host=127.0.0.1:7116 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/rados-striper/2 --osd-journal=td/rados-striper/2/journal --chdir= --run-dir=td/rados-striper '--admin-socket=/tmp/ceph-asok.50563/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/rados-striper/$name.log' '--pid-file=td/rados-striper/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:18.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:18.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:18.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:18.458 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:18.450+0000 7f462e51ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:18.459 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:18.454+0000 7f462e51ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:18.460 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:18.454+0000 7f462e51ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:18.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-25T22:28:18.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:18.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-25T22:28:18.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:18.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:18.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:18.548 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:18.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:18.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:18.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:28:18.654 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:18.865 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:18.858+0000 7f462e51ea40 -1 Falling back to public interface 2026-02-25T22:28:19.448 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:19.442+0000 7f462e51ea40 -1 osd.2 0 log_to_monitors true 2026-02-25T22:28:19.655 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:19.656 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:19.656 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:28:19.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:19.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:19.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:28:19.766 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:20.767 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:20.768 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:20.768 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:20.768 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:28:20.768 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:20.769 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-25T22:28:20.873 INFO:tasks.workunit.client.0.trial188.stdout:osd.2 up in weight 1 up_from 14 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6816/66643335,v1:127.0.0.1:6817/66643335] [v2:127.0.0.1:6818/66643335,v1:127.0.0.1:6819/66643335] exists,up 1f70bffb-a264-4e19-9b82-833a503939cc 2026-02-25T22:28:20.874 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:20.874 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:20.874 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:20.874 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:36: run: create_rbd_pool 2026-02-25T22:28:20.874 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:28:20.973 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:28:20.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:28:20.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:28:21.120 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:28:21.128 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:28:22.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:28:22.407 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:39: run: dd if=/dev/urandom of=td/rados-striper/toyfile bs=1234 count=1 2026-02-25T22:28:22.408 INFO:tasks.workunit.client.0.trial188.stderr:1+0 records in 2026-02-25T22:28:22.408 INFO:tasks.workunit.client.0.trial188.stderr:1+0 records out 2026-02-25T22:28:22.408 INFO:tasks.workunit.client.0.trial188.stderr:1234 bytes (1.2 kB, 1.2 KiB) copied, 3.2201e-05 s, 38.3 MB/s 2026-02-25T22:28:22.408 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:42: run: rados --pool rbd --striper put toyfile td/rados-striper/toyfile 2026-02-25T22:28:22.435 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: rados --pool rbd --striper stat toyfile 2026-02-25T22:28:22.435 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:45: run: cut -d , -f 2 2026-02-25T22:28:22.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: rados --pool rbd stat toyfile.0000000000000000 2026-02-25T22:28:22.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:46: run: cut -d , -f 2 2026-02-25T22:28:22.480 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:47: run: echo ' size 1234' 2026-02-25T22:28:22.481 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:48: run: diff -w td/rados-striper/stripedStat td/rados-striper/refstat 2026-02-25T22:28:22.482 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:49: run: diff -w td/rados-striper/stat td/rados-striper/refstat 2026-02-25T22:28:22.483 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:50: run: rados --pool rbd stat toyfile 2026-02-25T22:28:22.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:51: run: grep -q 'No such file or directory' td/rados-striper/staterror 2026-02-25T22:28:22.508 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:54: run: rados --pool rbd --striper get toyfile td/rados-striper/stripedGroup 2026-02-25T22:28:22.533 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:55: run: diff -w td/rados-striper/toyfile td/rados-striper/stripedGroup 2026-02-25T22:28:22.534 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:56: run: rados --pool rbd get toyfile.0000000000000000 td/rados-striper/nonSTripedGroup 2026-02-25T22:28:22.556 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:57: run: diff -w td/rados-striper/toyfile td/rados-striper/nonSTripedGroup 2026-02-25T22:28:22.558 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:60: run: rados --pool rbd --striper truncate toyfile 12 2026-02-25T22:28:22.584 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: rados --pool rbd --striper stat toyfile 2026-02-25T22:28:22.584 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:61: run: cut -d , -f 2 2026-02-25T22:28:22.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: rados --pool rbd stat toyfile.0000000000000000 2026-02-25T22:28:22.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:62: run: cut -d , -f 2 2026-02-25T22:28:22.617 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:63: run: echo ' size 12' 2026-02-25T22:28:22.617 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:64: run: diff -w td/rados-striper/stripedStat td/rados-striper/reftrunc 2026-02-25T22:28:22.618 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:65: run: diff -w td/rados-striper/stat td/rados-striper/reftrunc 2026-02-25T22:28:22.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:69: run: rados --pool rbd --striper setxattr toyfile somexattr somevalue 2026-02-25T22:28:22.644 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:70: run: rados --pool rbd --striper getxattr toyfile somexattr 2026-02-25T22:28:22.668 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:71: run: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-25T22:28:22.692 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:72: run: echo somevalue 2026-02-25T22:28:22.692 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:73: run: diff -w td/rados-striper/xattrvalue td/rados-striper/refvalue 2026-02-25T22:28:22.693 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:74: run: diff -w td/rados-striper/xattrvalue2 td/rados-striper/refvalue 2026-02-25T22:28:22.695 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:75: run: rados --pool rbd --striper listxattr toyfile 2026-02-25T22:28:22.718 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:76: run: echo somexattr 2026-02-25T22:28:22.718 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:77: run: diff -w td/rados-striper/xattrlist td/rados-striper/reflist 2026-02-25T22:28:22.720 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: rados --pool rbd listxattr toyfile.0000000000000000 2026-02-25T22:28:22.720 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:78: run: grep -v striper 2026-02-25T22:28:22.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:79: run: diff -w td/rados-striper/xattrlist2 td/rados-striper/reflist 2026-02-25T22:28:22.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:80: run: rados --pool rbd --striper rmxattr toyfile somexattr 2026-02-25T22:28:22.764 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:82: run: local 'attr_not_found_str=No data available' 2026-02-25T22:28:22.765 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: uname 2026-02-25T22:28:22.766 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:83: run: '[' Linux = FreeBSD ']' 2026-02-25T22:28:22.766 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:85: run: expect_failure td/rados-striper 'No data available' rados --pool rbd --striper getxattr toyfile somexattr 2026-02-25T22:28:22.767 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/rados-striper 2026-02-25T22:28:22.767 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:28:22.767 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local 'expected=No data available' 2026-02-25T22:28:22.767 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:28:22.767 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:28:22.767 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: rados --pool rbd --striper getxattr toyfile somexattr 2026-02-25T22:28:22.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-25T22:28:22.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-25T22:28:22.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2026-02-25T22:28:22.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-25T22:28:22.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:87: run: expect_failure td/rados-striper 'No data available' rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-25T22:28:22.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/rados-striper 2026-02-25T22:28:22.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:28:22.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local 'expected=No data available' 2026-02-25T22:28:22.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:28:22.787 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:28:22.787 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: rados --pool rbd getxattr toyfile.0000000000000000 somexattr 2026-02-25T22:28:22.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-25T22:28:22.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-25T22:28:22.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet 'No data available' td/rados-striper/out 2026-02-25T22:28:22.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-25T22:28:22.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:91: run: rados --pool rbd --striper rm toyfile 2026-02-25T22:28:22.823 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:92: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd --striper stat toyfile 2026-02-25T22:28:22.823 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/rados-striper 2026-02-25T22:28:22.823 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:28:22.823 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local 'expected=No such file or directory' 2026-02-25T22:28:22.823 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:28:22.823 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:28:22.823 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: rados --pool rbd --striper stat toyfile 2026-02-25T22:28:22.846 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-25T22:28:22.846 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-25T22:28:22.846 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2026-02-25T22:28:22.848 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-25T22:28:22.848 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:94: run: expect_failure td/rados-striper 'No such file or directory' rados --pool rbd stat toyfile.0000000000000000 2026-02-25T22:28:22.848 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/rados-striper 2026-02-25T22:28:22.848 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:28:22.848 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local 'expected=No such file or directory' 2026-02-25T22:28:22.848 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:28:22.849 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:28:22.849 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: rados --pool rbd stat toyfile.0000000000000000 2026-02-25T22:28:22.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-25T22:28:22.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-25T22:28:22.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet 'No such file or directory' td/rados-striper/out 2026-02-25T22:28:22.870 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-25T22:28:22.870 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/rados-striper.sh:98: run: teardown td/rados-striper 2026-02-25T22:28:22.870 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2026-02-25T22:28:22.870 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:22.870 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2026-02-25T22:28:22.871 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:22.871 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:22.871 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:22.871 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:22.871 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:22.979 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:22.980 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:22.981 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:22.981 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:22.982 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:22.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:22.983 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:22.984 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:22.984 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:22.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:22.985 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:22.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:22.986 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:22.986 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2026-02-25T22:28:22.996 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:22.996 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:22.996 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50563 2026-02-25T22:28:22.996 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.50563 2026-02-25T22:28:22.998 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:22.998 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:22.998 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: code=0 2026-02-25T22:28:22.999 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2404: main: teardown td/rados-striper 0 2026-02-25T22:28:22.999 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/rados-striper 2026-02-25T22:28:22.999 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-25T22:28:22.999 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/rados-striper KILL 2026-02-25T22:28:22.999 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:22.999 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:22.999 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:22.999 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:22.999 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:23.000 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:23.001 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:23.001 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:23.002 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:23.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:23.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:23.004 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:23.005 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:23.005 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:23.005 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:23.006 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:23.007 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:23.008 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-25T22:28:23.008 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/rados-striper 2026-02-25T22:28:23.009 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:23.009 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:23.009 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.50563 2026-02-25T22:28:23.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.50563 2026-02-25T22:28:23.010 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:23.010 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:23.010 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: return 0 2026-02-25T22:28:23.011 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-25T22:28:23.012 DEBUG:teuthology.orchestra.run.trial188:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-25T22:28:23.064 INFO:tasks.workunit:Running workunit misc/test-ceph-helpers.sh... 2026-02-25T22:28:23.064 DEBUG:teuthology.orchestra.run.trial188:workunit test misc/test-ceph-helpers.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61d4431d3ec879bd98659388b3655acf3d392418 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh 2026-02-25T22:28:23.117 INFO:tasks.workunit.client.0.trial188.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-25T22:28:23.120 INFO:tasks.workunit.client.0.trial188.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-25T22:28:23.121 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: run_tests: export PATH=./bin:.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:28:23.121 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2414: run_tests: PATH=./bin:.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-25T22:28:23.121 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: run_tests: export CEPH_MON=127.0.0.1:7109 2026-02-25T22:28:23.121 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2416: run_tests: CEPH_MON=127.0.0.1:7109 2026-02-25T22:28:23.121 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2417: run_tests: export CEPH_ARGS 2026-02-25T22:28:23.121 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: run_tests: uuidgen 2026-02-25T22:28:23.122 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2418: run_tests: CEPH_ARGS+=' --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none ' 2026-02-25T22:28:23.122 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2419: run_tests: CEPH_ARGS+='--mon-host=127.0.0.1:7109 ' 2026-02-25T22:28:23.123 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: run_tests: export CEPH_CONF=/dev/null 2026-02-25T22:28:23.123 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2420: run_tests: CEPH_CONF=/dev/null 2026-02-25T22:28:23.123 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: run_tests: set 2026-02-25T22:28:23.123 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: run_tests: sed -n -e 's/^\(test_[0-9a-z_]*\) .*/\1/p' 2026-02-25T22:28:23.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2422: run_tests: local 'funcs=test_activate_osd 2026-02-25T22:28:23.125 INFO:tasks.workunit.client.0.trial188.stderr:test_activate_osd_after_mark_down 2026-02-25T22:28:23.125 INFO:tasks.workunit.client.0.trial188.stderr:test_activate_osd_skip_benchmark 2026-02-25T22:28:23.125 INFO:tasks.workunit.client.0.trial188.stderr:test_destroy_osd 2026-02-25T22:28:23.125 INFO:tasks.workunit.client.0.trial188.stderr:test_display_logs 2026-02-25T22:28:23.126 INFO:tasks.workunit.client.0.trial188.stderr:test_erasure_code_plugin_exists 2026-02-25T22:28:23.126 INFO:tasks.workunit.client.0.trial188.stderr:test_expect_failure 2026-02-25T22:28:23.126 INFO:tasks.workunit.client.0.trial188.stderr:test_file_with_random_data 2026-02-25T22:28:23.126 INFO:tasks.workunit.client.0.trial188.stderr:test_flush_pg_stats 2026-02-25T22:28:23.126 INFO:tasks.workunit.client.0.trial188.stderr:test_get_config 2026-02-25T22:28:23.126 INFO:tasks.workunit.client.0.trial188.stderr:test_get_is_making_recovery_progress 2026-02-25T22:28:23.126 INFO:tasks.workunit.client.0.trial188.stderr:test_get_last_scrub_stamp 2026-02-25T22:28:23.126 INFO:tasks.workunit.client.0.trial188.stderr:test_get_not_primary 2026-02-25T22:28:23.126 INFO:tasks.workunit.client.0.trial188.stderr:test_get_num_active_clean 2026-02-25T22:28:23.126 INFO:tasks.workunit.client.0.trial188.stderr:test_get_num_active_or_peered 2026-02-25T22:28:23.126 INFO:tasks.workunit.client.0.trial188.stderr:test_get_num_pgs 2026-02-25T22:28:23.126 INFO:tasks.workunit.client.0.trial188.stderr:test_get_op_scheduler 2026-02-25T22:28:23.126 INFO:tasks.workunit.client.0.trial188.stderr:test_get_osd_id_used_by_pgs 2026-02-25T22:28:23.127 INFO:tasks.workunit.client.0.trial188.stderr:test_get_osds 2026-02-25T22:28:23.127 INFO:tasks.workunit.client.0.trial188.stderr:test_get_pg 2026-02-25T22:28:23.127 INFO:tasks.workunit.client.0.trial188.stderr:test_get_primary 2026-02-25T22:28:23.127 INFO:tasks.workunit.client.0.trial188.stderr:test_get_timeout_delays 2026-02-25T22:28:23.127 INFO:tasks.workunit.client.0.trial188.stderr:test_is_clean 2026-02-25T22:28:23.127 INFO:tasks.workunit.client.0.trial188.stderr:test_kill_daemon 2026-02-25T22:28:23.127 INFO:tasks.workunit.client.0.trial188.stderr:test_kill_daemons 2026-02-25T22:28:23.127 INFO:tasks.workunit.client.0.trial188.stderr:test_objectstore_tool 2026-02-25T22:28:23.127 INFO:tasks.workunit.client.0.trial188.stderr:test_pg_schedule_scrub 2026-02-25T22:28:23.127 INFO:tasks.workunit.client.0.trial188.stderr:test_pg_scrub 2026-02-25T22:28:23.127 INFO:tasks.workunit.client.0.trial188.stderr:test_repair 2026-02-25T22:28:23.127 INFO:tasks.workunit.client.0.trial188.stderr:test_run_in_background 2026-02-25T22:28:23.127 INFO:tasks.workunit.client.0.trial188.stderr:test_run_mon 2026-02-25T22:28:23.128 INFO:tasks.workunit.client.0.trial188.stderr:test_run_osd 2026-02-25T22:28:23.128 INFO:tasks.workunit.client.0.trial188.stderr:test_set_config 2026-02-25T22:28:23.128 INFO:tasks.workunit.client.0.trial188.stderr:test_setup 2026-02-25T22:28:23.128 INFO:tasks.workunit.client.0.trial188.stderr:test_teardown 2026-02-25T22:28:23.128 INFO:tasks.workunit.client.0.trial188.stderr:test_wait_background 2026-02-25T22:28:23.128 INFO:tasks.workunit.client.0.trial188.stderr:test_wait_for_clean 2026-02-25T22:28:23.128 INFO:tasks.workunit.client.0.trial188.stderr:test_wait_for_health_ok 2026-02-25T22:28:23.128 INFO:tasks.workunit.client.0.trial188.stderr:test_wait_for_osd 2026-02-25T22:28:23.128 INFO:tasks.workunit.client.0.trial188.stderr:test_wait_for_peered 2026-02-25T22:28:23.128 INFO:tasks.workunit.client.0.trial188.stderr:test_wait_for_scrub 2026-02-25T22:28:23.128 INFO:tasks.workunit.client.0.trial188.stderr:test_wait_osd_id_used_by_pgs' 2026-02-25T22:28:23.128 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2423: run_tests: local dir=td/ceph-helpers 2026-02-25T22:28:23.128 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:28:23.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_activate_osd td/ceph-helpers 2026-02-25T22:28:23.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:877: test_activate_osd: local dir=td/ceph-helpers 2026-02-25T22:28:23.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:879: test_activate_osd: setup td/ceph-helpers 2026-02-25T22:28:23.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:28:23.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:28:23.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:23.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:23.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:23.129 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:23.129 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:23.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:23.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:23.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:23.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:23.131 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:23.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:23.131 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:23.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:23.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:23.131 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:23.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:23.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:23.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:23.132 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:23.132 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:23.133 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:23.133 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:23.135 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:23.135 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:23.135 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:23.135 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:23.136 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:23.136 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:23.136 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:28:23.138 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:28:23.138 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:23.138 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:23.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:28:23.140 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:28:23.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-25T22:28:23.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-25T22:28:23.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:28:23.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:28:23.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:881: test_activate_osd: run_mon td/ceph-helpers a 2026-02-25T22:28:23.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:28:23.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:28:23.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:28:23.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:28:23.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:28:23.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:28:23.166 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:28:23.166 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:23.166 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:23.166 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:23.166 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:23.166 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:23.167 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:23.167 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:28:23.195 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:28:23.196 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:28:23.197 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:23.197 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:23.197 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:28:23.198 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:28:23.198 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:23.198 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:23.198 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:23.198 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:23.198 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:23.199 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:23.199 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:23.199 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:23.199 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:28:23.254 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:28:23.254 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:23.255 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:23.255 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:28:23.256 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:28:23.256 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:23.256 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:23.256 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:23.256 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:23.256 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:23.256 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:23.256 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:23.257 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:23.257 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:28:23.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:882: test_activate_osd: run_mgr td/ceph-helpers x 2026-02-25T22:28:23.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:28:23.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:28:23.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:28:23.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:28:23.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:28:23.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:28:23.424 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:28:23.424 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:23.424 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:23.424 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:23.424 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:23.424 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:23.424 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:23.425 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:23.426 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:23.445 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:884: test_activate_osd: run_osd td/ceph-helpers 0 2026-02-25T22:28:23.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:28:23.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:23.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:28:23.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:23.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:28:23.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:28:23.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:23.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:23.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:23.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:28:23.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:28:23.447 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:23.447 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:23.447 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:28:23.447 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:23.447 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:23.447 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:23.447 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:23.447 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:23.447 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:23.447 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:23.447 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:23.447 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:23.448 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:23.448 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:23.448 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:28:23.448 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:28:23.448 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:23.448 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:23.448 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:23.448 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:23.448 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:23.448 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:23.448 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:28:23.449 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:23.449 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=657eccc0-2236-4bde-85e3-e7f5c81f1c83 2026-02-25T22:28:23.449 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 657eccc0-2236-4bde-85e3-e7f5c81f1c83' 2026-02-25T22:28:23.450 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 657eccc0-2236-4bde-85e3-e7f5c81f1c83 2026-02-25T22:28:23.450 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:23.459 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCHd59pfdI/GxAAfHULRvAr8YyHcPwgosTBkg== 2026-02-25T22:28:23.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCHd59pfdI/GxAAfHULRvAr8YyHcPwgosTBkg=="}' 2026-02-25T22:28:23.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 657eccc0-2236-4bde-85e3-e7f5c81f1c83 -i td/ceph-helpers/0/new.json 2026-02-25T22:28:23.573 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:23.584 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:28:23.585 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCHd59pfdI/GxAAfHULRvAr8YyHcPwgosTBkg== --osd-uuid 657eccc0-2236-4bde-85e3-e7f5c81f1c83 2026-02-25T22:28:23.601 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:23.594+0000 7fb6b1f62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:23.603 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:23.594+0000 7fb6b1f62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:23.603 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:23.598+0000 7fb6b1f62a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:23.604 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:23.598+0000 7fb6b1f62a40 -1 bdev(0x5607a7700400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:23.604 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:23.598+0000 7fb6b1f62a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:28:24.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:28:24.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:24.388 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:28:24.388 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:28:24.388 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:24.499 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:28:24.500 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:28:24.500 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:24.500 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:24.500 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:24.500 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:24.515 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:24.506+0000 7f99a12e4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:24.516 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:24.510+0000 7f99a12e4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:24.517 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:24.510+0000 7f99a12e4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:24.617 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:28:24.617 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:24.617 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:24.617 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:24.617 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:24.618 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:24.618 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:24.618 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:24.618 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:24.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:24.729 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:24.937 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:24.930+0000 7f99a12e4a40 -1 Falling back to public interface 2026-02-25T22:28:25.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:25.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:25.731 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:28:25.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:25.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:25.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:25.842 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:25.959 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:25.954+0000 7f99a12e4a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:28:26.843 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:26.844 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:26.844 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:28:26.845 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:26.845 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:26.845 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:27.058 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1928290646,v1:127.0.0.1:6801/1928290646] [v2:127.0.0.1:6802/1928290646,v1:127.0.0.1:6803/1928290646] exists,up 657eccc0-2236-4bde-85e3-e7f5c81f1c83 2026-02-25T22:28:27.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:27.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:27.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:27.060 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: get_asok_path osd.0 2026-02-25T22:28:27.060 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:28:27.061 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:28:27.061 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:27.061 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:27.061 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:27.061 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-osd.0.asok 2026-02-25T22:28:27.061 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: CEPH_ARGS= 2026-02-25T22:28:27.061 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.55572/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T22:28:27.122 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:886: test_activate_osd: local 'backfills={"osd_max_backfills":"1"}' 2026-02-25T22:28:27.122 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:887: test_activate_osd: echo '{"osd_max_backfills":"1"}' 2026-02-25T22:28:27.122 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:887: test_activate_osd: grep --quiet osd_max_backfills 2026-02-25T22:28:27.124 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:889: test_activate_osd: kill_daemons td/ceph-helpers TERM osd 2026-02-25T22:28:27.124 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:27.124 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:27.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:27.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:27.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:27.433 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:27.433 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:891: test_activate_osd: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2026-02-25T22:28:27.433 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-25T22:28:27.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:28:27.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-25T22:28:27.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:28:27.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:28:27.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:28:27.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:27.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:27.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:27.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:28:27.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:28:27.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:27.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:28:27.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:28:27.435 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:28:27.435 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:27.435 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:27.436 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:27.436 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:27.436 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:27.436 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:27.436 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:27.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:27.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:28:27.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:28:27.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:27.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:27.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:27.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:27.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:28:27.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-max-backfills 20' 2026-02-25T22:28:27.437 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:28:27.438 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:28:27.438 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-25T22:28:27.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2026-02-25T22:28:27.439 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-25T22:28:27.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-25T22:28:27.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:28:27.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:28:27.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:28:27.459 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:27.454+0000 7f474d28aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:27.461 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:27.454+0000 7f474d28aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:27.462 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:27.454+0000 7f474d28aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:27.645 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-25T22:28:27.645 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:27.645 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:27.645 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:27.645 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:27.645 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:27.646 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:27.646 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:27.646 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:27.647 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:27.829 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:27.822+0000 7f474d28aa40 -1 Falling back to public interface 2026-02-25T22:28:27.864 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:28.482 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:28.474+0000 7f474d28aa40 -1 osd.0 5 log_to_monitors true 2026-02-25T22:28:28.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:28.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:28.866 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:28:28.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:28.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:28.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:29.085 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 8 up_thru 0 down_at 6 last_clean_interval [0,0) [v2:127.0.0.1:6800/3201268340,v1:127.0.0.1:6801/3201268340] [v2:127.0.0.1:6802/3201268340,v1:127.0.0.1:6803/3201268340] exists,up 657eccc0-2236-4bde-85e3-e7f5c81f1c83 2026-02-25T22:28:29.085 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:29.085 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:29.085 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:29.086 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: get_asok_path osd.0 2026-02-25T22:28:29.087 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:28:29.087 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:28:29.087 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:29.087 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:29.087 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:29.087 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-osd.0.asok 2026-02-25T22:28:29.087 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: CEPH_ARGS= 2026-02-25T22:28:29.087 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: ceph --format=json daemon /tmp/ceph-asok.55572/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T22:28:29.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:893: test_activate_osd: local 'backfills={"osd_max_backfills":"20"}' 2026-02-25T22:28:29.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:894: test_activate_osd: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2026-02-25T22:28:29.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:896: test_activate_osd: teardown td/ceph-helpers 2026-02-25T22:28:29.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:29.149 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:29.149 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:29.149 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:29.149 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:29.149 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:29.149 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:29.149 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:29.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:29.256 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:29.257 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:29.257 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:29.258 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:29.258 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:29.258 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:29.259 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:29.259 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:29.260 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:29.260 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:29.261 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:29.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:29.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:29.267 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:29.268 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:29.268 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:29.268 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:29.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:29.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:29.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:28:29.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_activate_osd_after_mark_down td/ceph-helpers 2026-02-25T22:28:29.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:900: test_activate_osd_after_mark_down: local dir=td/ceph-helpers 2026-02-25T22:28:29.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:902: test_activate_osd_after_mark_down: setup td/ceph-helpers 2026-02-25T22:28:29.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:28:29.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:28:29.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:29.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:29.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:29.271 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:29.271 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:29.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:29.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:29.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:29.272 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:29.272 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:29.273 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:29.273 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:29.274 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:29.274 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:29.274 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:29.275 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:29.276 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:29.276 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:29.276 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:29.277 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:29.278 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:29.278 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:29.279 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:29.279 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:29.280 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:29.280 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:29.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:29.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:29.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:28:29.282 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:28:29.282 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:29.282 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:29.282 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:28:29.284 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:28:29.285 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:28:29.285 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:28:29.285 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:28:29.285 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:904: test_activate_osd_after_mark_down: run_mon td/ceph-helpers a 2026-02-25T22:28:29.285 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:28:29.285 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:28:29.285 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:28:29.285 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:28:29.286 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:28:29.286 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:28:29.307 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:28:29.308 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:29.308 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:29.308 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:29.308 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:29.308 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:29.308 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:29.309 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:28:29.339 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:28:29.340 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:28:29.341 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:29.341 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:29.341 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:28:29.342 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:28:29.342 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:29.342 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:29.342 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:29.342 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:29.342 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:29.342 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:29.343 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:29.343 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:29.343 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:28:29.399 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:28:29.399 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:29.399 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:29.399 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:28:29.399 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:28:29.400 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:29.400 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:29.400 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:29.401 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:29.401 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:29.401 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:29.401 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:29.401 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:29.401 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:28:29.456 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:905: test_activate_osd_after_mark_down: run_mgr td/ceph-helpers x 2026-02-25T22:28:29.456 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:28:29.456 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:28:29.456 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:28:29.457 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:28:29.457 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:28:29.457 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:28:29.570 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:28:29.570 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:29.571 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:29.571 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:29.572 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:29.572 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:29.572 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:29.572 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:29.572 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:29.591 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:907: test_activate_osd_after_mark_down: run_osd td/ceph-helpers 0 2026-02-25T22:28:29.591 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:28:29.591 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:29.592 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:28:29.592 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:29.592 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:28:29.592 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:28:29.592 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:29.592 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:29.592 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:29.592 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:28:29.592 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:28:29.592 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:29.592 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:29.592 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:28:29.593 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:29.593 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:29.593 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:29.593 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:29.593 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:29.593 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:29.593 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:29.593 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:29.593 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:29.593 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:29.593 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:29.593 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:28:29.594 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:28:29.594 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:29.594 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:29.594 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:29.594 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:29.594 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:29.594 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:29.594 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:28:29.595 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:29.595 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e0b38220-db25-42aa-9f46-407f1e80af0f 2026-02-25T22:28:29.595 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e0b38220-db25-42aa-9f46-407f1e80af0f' 2026-02-25T22:28:29.595 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 e0b38220-db25-42aa-9f46-407f1e80af0f 2026-02-25T22:28:29.595 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:29.605 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCNd59plIjtIxAA6igGoF30SE4gljToSlj5eg== 2026-02-25T22:28:29.605 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCNd59plIjtIxAA6igGoF30SE4gljToSlj5eg=="}' 2026-02-25T22:28:29.605 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e0b38220-db25-42aa-9f46-407f1e80af0f -i td/ceph-helpers/0/new.json 2026-02-25T22:28:29.715 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:29.723 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:28:29.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCNd59plIjtIxAA6igGoF30SE4gljToSlj5eg== --osd-uuid e0b38220-db25-42aa-9f46-407f1e80af0f 2026-02-25T22:28:29.740 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:29.734+0000 7f14c8c80a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:29.742 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:29.734+0000 7f14c8c80a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:29.742 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:29.734+0000 7f14c8c80a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:29.743 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:29.738+0000 7f14c8c80a40 -1 bdev(0x55c8e2048400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:29.743 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:29.738+0000 7f14c8c80a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:28:30.451 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:28:30.451 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:30.453 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:28:30.453 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:28:30.453 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:30.572 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:28:30.573 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:28:30.573 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:30.573 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:30.573 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:30.573 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:30.595 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:30.586+0000 7f537215ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:30.597 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:30.590+0000 7f537215ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:30.598 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:30.590+0000 7f537215ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:30.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:28:30.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:30.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:30.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:30.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:30.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:30.686 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:30.686 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:30.686 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:30.686 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:30.794 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:31.089 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:31.082+0000 7f537215ea40 -1 Falling back to public interface 2026-02-25T22:28:31.796 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:31.796 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:31.796 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:28:31.797 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:31.797 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:31.797 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:31.906 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:32.082 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:32.074+0000 7f537215ea40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:28:32.907 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:32.908 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:32.908 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:28:32.909 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:32.909 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:32.909 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:33.126 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/811929926,v1:127.0.0.1:6801/811929926] [v2:127.0.0.1:6802/811929926,v1:127.0.0.1:6803/811929926] exists,up e0b38220-db25-42aa-9f46-407f1e80af0f 2026-02-25T22:28:33.126 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:33.126 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:33.126 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:33.127 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: get_asok_path osd.0 2026-02-25T22:28:33.127 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:28:33.127 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:28:33.127 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:33.128 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:33.128 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:33.128 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-osd.0.asok 2026-02-25T22:28:33.128 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: CEPH_ARGS= 2026-02-25T22:28:33.128 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.55572/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T22:28:33.187 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:909: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"1"}' 2026-02-25T22:28:33.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:910: test_activate_osd_after_mark_down: echo '{"osd_max_backfills":"1"}' 2026-02-25T22:28:33.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:910: test_activate_osd_after_mark_down: grep --quiet osd_max_backfills 2026-02-25T22:28:33.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:912: test_activate_osd_after_mark_down: kill_daemons td/ceph-helpers TERM osd 2026-02-25T22:28:33.190 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:33.190 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:33.190 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:33.190 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:33.190 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:33.498 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:33.498 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:913: test_activate_osd_after_mark_down: ceph osd down 0 2026-02-25T22:28:33.714 INFO:tasks.workunit.client.0.trial188.stderr:osd.0 is already down. 2026-02-25T22:28:33.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:914: test_activate_osd_after_mark_down: wait_for_osd down 0 2026-02-25T22:28:33.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2026-02-25T22:28:33.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:33.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:33.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:33.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:33.725 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:33.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:33.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:33.726 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2026-02-25T22:28:33.938 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6800/811929926,v1:127.0.0.1:6801/811929926] [v2:127.0.0.1:6802/811929926,v1:127.0.0.1:6803/811929926] exists e0b38220-db25-42aa-9f46-407f1e80af0f 2026-02-25T22:28:33.941 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:33.941 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:33.941 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:33.941 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:916: test_activate_osd_after_mark_down: activate_osd td/ceph-helpers 0 --osd-max-backfills 20 2026-02-25T22:28:33.941 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-25T22:28:33.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:28:33.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-25T22:28:33.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:28:33.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:28:33.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:28:33.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:33.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:33.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:33.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:28:33.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:28:33.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:33.943 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:28:33.943 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:28:33.943 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:28:33.943 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:33.943 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:33.943 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:33.943 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:33.943 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:33.943 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:33.943 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:33.943 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:33.943 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:28:33.943 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:28:33.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:33.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:33.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:33.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:33.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:28:33.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-max-backfills 20' 2026-02-25T22:28:33.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:28:33.945 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:28:33.946 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-25T22:28:33.946 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-max-backfills 20 2026-02-25T22:28:33.946 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-25T22:28:33.946 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-25T22:28:33.946 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:28:33.946 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:28:33.946 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:28:33.965 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:33.958+0000 7fbd68e76a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:33.967 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:33.958+0000 7fbd68e76a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:33.968 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:33.962+0000 7fbd68e76a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:34.155 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-25T22:28:34.155 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:34.155 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:34.156 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:34.156 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:34.156 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:34.156 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:34.157 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:34.157 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:34.157 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:34.313 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:34.306+0000 7fbd68e76a40 -1 Falling back to public interface 2026-02-25T22:28:34.379 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:35.026 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:35.018+0000 7fbd68e76a40 -1 osd.0 6 log_to_monitors true 2026-02-25T22:28:35.380 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:35.380 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:35.381 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:28:35.381 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:35.381 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:35.381 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:35.597 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6800/3519883678,v1:127.0.0.1:6801/3519883678] [v2:127.0.0.1:6802/3519883678,v1:127.0.0.1:6803/3519883678] exists,up e0b38220-db25-42aa-9f46-407f1e80af0f 2026-02-25T22:28:35.597 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:35.598 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:35.598 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:35.599 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: get_asok_path osd.0 2026-02-25T22:28:35.599 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:28:35.599 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:28:35.599 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:35.599 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:35.599 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:35.599 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-osd.0.asok 2026-02-25T22:28:35.599 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: CEPH_ARGS= 2026-02-25T22:28:35.600 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: ceph --format=json daemon /tmp/ceph-asok.55572/ceph-osd.0.asok config get osd_max_backfills 2026-02-25T22:28:35.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:918: test_activate_osd_after_mark_down: local 'backfills={"osd_max_backfills":"20"}' 2026-02-25T22:28:35.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:919: test_activate_osd_after_mark_down: test '{"osd_max_backfills":"20"}' = '{"osd_max_backfills":"20"}' 2026-02-25T22:28:35.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:921: test_activate_osd_after_mark_down: teardown td/ceph-helpers 2026-02-25T22:28:35.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:35.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:35.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:35.660 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:35.660 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:35.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:35.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:35.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:35.768 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:35.768 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:35.770 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:35.770 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:35.771 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:35.771 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:35.772 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:35.773 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:35.773 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:35.773 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:35.773 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:35.774 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:35.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:35.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:35.781 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:35.781 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:35.781 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:35.781 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:35.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:35.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:35.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:28:35.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_activate_osd_skip_benchmark td/ceph-helpers 2026-02-25T22:28:35.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:925: test_activate_osd_skip_benchmark: local dir=td/ceph-helpers 2026-02-25T22:28:35.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:927: test_activate_osd_skip_benchmark: setup td/ceph-helpers 2026-02-25T22:28:35.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:28:35.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:28:35.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:35.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:35.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:35.785 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:35.785 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:35.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:35.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:35.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:35.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:35.786 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:35.787 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:35.787 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:35.789 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:35.789 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:35.789 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:35.790 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:35.790 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:35.791 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:35.791 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:35.792 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:35.792 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:35.793 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:35.794 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:35.794 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:35.794 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:35.794 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:35.795 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:35.795 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:35.795 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:28:35.797 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:28:35.797 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:35.797 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:35.797 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:28:35.799 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:28:35.800 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:28:35.800 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:28:35.800 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:28:35.800 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:929: test_activate_osd_skip_benchmark: run_mon td/ceph-helpers a 2026-02-25T22:28:35.800 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:28:35.800 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:28:35.800 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:28:35.800 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:28:35.800 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:28:35.801 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:28:35.827 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:28:35.828 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:35.828 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:35.828 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:35.828 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:35.828 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:35.828 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:35.829 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:28:35.851 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:28:35.852 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:28:35.852 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:35.852 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:35.852 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:28:35.853 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:28:35.853 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:35.854 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:35.854 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:35.854 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:35.854 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:35.854 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:35.854 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:35.854 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:35.854 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:28:35.909 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:28:35.909 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:35.910 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:35.910 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:28:35.910 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:28:35.910 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:35.910 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:35.910 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:35.911 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:35.911 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:35.911 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:35.911 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:35.911 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:35.911 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:28:35.964 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:930: test_activate_osd_skip_benchmark: run_mgr td/ceph-helpers x 2026-02-25T22:28:35.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:28:35.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:28:35.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:28:35.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:28:35.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:28:35.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:28:36.075 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:28:36.075 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:36.075 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:36.076 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:36.076 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:36.076 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:36.076 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:36.076 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:36.077 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:36.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:933: test_activate_osd_skip_benchmark: run_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-25T22:28:36.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:28:36.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:36.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:28:36.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:36.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:28:36.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:28:36.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:36.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:36.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:36.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:28:36.100 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:28:36.100 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:36.100 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:36.100 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:28:36.100 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:36.100 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:36.100 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:36.100 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:36.100 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:36.100 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:36.100 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:36.100 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:36.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:36.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:36.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:36.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:28:36.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:28:36.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:36.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:36.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:36.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:36.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:36.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2026-02-25T22:28:36.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:28:36.102 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:36.102 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=e78cc6b4-a349-4f11-a6e4-98daa2c7e8c8 2026-02-25T22:28:36.102 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 e78cc6b4-a349-4f11-a6e4-98daa2c7e8c8 2026-02-25T22:28:36.103 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 e78cc6b4-a349-4f11-a6e4-98daa2c7e8c8' 2026-02-25T22:28:36.103 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:36.112 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCUd59pjQuTBhAA4wT9CWwyeRdjls23S999Ww== 2026-02-25T22:28:36.113 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCUd59pjQuTBhAA4wT9CWwyeRdjls23S999Ww=="}' 2026-02-25T22:28:36.113 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new e78cc6b4-a349-4f11-a6e4-98daa2c7e8c8 -i td/ceph-helpers/0/new.json 2026-02-25T22:28:36.227 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:36.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:28:36.237 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true --mkfs --key AQCUd59pjQuTBhAA4wT9CWwyeRdjls23S999Ww== --osd-uuid e78cc6b4-a349-4f11-a6e4-98daa2c7e8c8 2026-02-25T22:28:36.253 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:36.246+0000 7efcde416a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:36.255 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:36.250+0000 7efcde416a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:36.256 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:36.250+0000 7efcde416a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:36.256 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:36.250+0000 7efcde416a40 -1 bdev(0x564d96fee400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:36.256 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:36.250+0000 7efcde416a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:28:37.042 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:28:37.042 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:37.044 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:28:37.044 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:28:37.044 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:37.165 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:28:37.166 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:28:37.166 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-25T22:28:37.166 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:37.167 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:37.167 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:37.191 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:37.182+0000 7ff1bc955a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:37.193 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:37.186+0000 7ff1bc955a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:37.194 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:37.186+0000 7ff1bc955a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:37.273 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:28:37.274 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:37.274 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:37.274 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:37.274 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:37.274 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:37.274 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:37.275 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:37.275 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:37.275 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:37.382 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:37.697 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:37.690+0000 7ff1bc955a40 -1 Falling back to public interface 2026-02-25T22:28:38.383 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:38.383 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:38.384 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:28:38.384 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:38.384 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:38.384 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:38.490 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:38.612 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:38.606+0000 7ff1bc955a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:28:39.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:39.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:39.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:39.493 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:28:39.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:39.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:39.700 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2955455935,v1:127.0.0.1:6801/2955455935] [v2:127.0.0.1:6802/2955455935,v1:127.0.0.1:6803/2955455935] exists,up e78cc6b4-a349-4f11-a6e4-98daa2c7e8c8 2026-02-25T22:28:39.701 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:39.701 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:39.701 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:39.702 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-25T22:28:39.702 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:28:39.702 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:28:39.703 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:39.703 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:39.703 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:39.703 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-osd.0.asok 2026-02-25T22:28:39.703 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-25T22:28:39.703 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.55572/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2026-02-25T22:28:39.760 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:936: test_activate_osd_skip_benchmark: local 'max_iops_hdd_def={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-25T22:28:39.761 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-25T22:28:39.761 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:28:39.761 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:28:39.762 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:39.762 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:39.762 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:39.762 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-osd.0.asok 2026-02-25T22:28:39.762 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-25T22:28:39.763 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.55572/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2026-02-25T22:28:39.825 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:938: test_activate_osd_skip_benchmark: local 'max_iops_ssd_def={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-25T22:28:39.825 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:940: test_activate_osd_skip_benchmark: kill_daemons td/ceph-helpers TERM osd 2026-02-25T22:28:39.826 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:39.826 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:39.826 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:39.826 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:39.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:40.134 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:40.134 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:941: test_activate_osd_skip_benchmark: ceph osd down 0 2026-02-25T22:28:40.348 INFO:tasks.workunit.client.0.trial188.stderr:osd.0 is already down. 2026-02-25T22:28:40.358 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:942: test_activate_osd_skip_benchmark: wait_for_osd down 0 2026-02-25T22:28:40.358 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=down 2026-02-25T22:28:40.358 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:40.358 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:40.358 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:40.358 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:40.359 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:40.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:40.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:40.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 down' 2026-02-25T22:28:40.593 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 down in weight 1 up_from 5 up_thru 0 down_at 7 last_clean_interval [0,0) [v2:127.0.0.1:6800/2955455935,v1:127.0.0.1:6801/2955455935] [v2:127.0.0.1:6802/2955455935,v1:127.0.0.1:6803/2955455935] exists e78cc6b4-a349-4f11-a6e4-98daa2c7e8c8 2026-02-25T22:28:40.596 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:40.596 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:40.596 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:40.597 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:946: test_activate_osd_skip_benchmark: activate_osd td/ceph-helpers 0 --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-25T22:28:40.597 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-25T22:28:40.597 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:28:40.597 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-25T22:28:40.597 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:28:40.597 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:28:40.597 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:28:40.597 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:40.597 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:40.597 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:40.597 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:28:40.597 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:28:40.598 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:40.598 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:28:40.598 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:28:40.598 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:28:40.598 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:40.598 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:40.598 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:40.598 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:40.598 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:40.598 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:40.598 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:40.598 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:40.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:28:40.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:28:40.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:40.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:40.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:40.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:40.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:28:40.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+='--osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true' 2026-02-25T22:28:40.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:28:40.599 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:28:40.600 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-25T22:28:40.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd-op-queue=mclock_scheduler --osd-mclock-skip-benchmark=true 2026-02-25T22:28:40.601 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-25T22:28:40.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-25T22:28:40.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:28:40.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:28:40.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:28:40.620 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:40.614+0000 7fe6c355da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:40.621 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:40.614+0000 7fe6c355da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:40.623 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:40.614+0000 7fe6c355da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:40.810 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-25T22:28:40.810 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:40.810 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:40.810 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:40.811 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:40.811 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:40.811 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:40.811 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:40.811 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:40.812 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:40.985 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:40.978+0000 7fe6c355da40 -1 Falling back to public interface 2026-02-25T22:28:41.029 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:41.571 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:41.566+0000 7fe6c355da40 -1 osd.0 6 log_to_monitors true 2026-02-25T22:28:42.030 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:42.030 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:42.031 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:28:42.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:42.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:42.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:42.243 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 9 up_thru 0 down_at 7 last_clean_interval [5,6) [v2:127.0.0.1:6800/421213645,v1:127.0.0.1:6801/421213645] [v2:127.0.0.1:6802/421213645,v1:127.0.0.1:6803/421213645] exists,up e78cc6b4-a349-4f11-a6e4-98daa2c7e8c8 2026-02-25T22:28:42.244 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:42.244 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:42.244 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:42.245 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-25T22:28:42.245 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:28:42.245 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:28:42.245 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:42.245 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:42.246 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:42.246 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-osd.0.asok 2026-02-25T22:28:42.246 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-25T22:28:42.246 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.55572/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_hdd 2026-02-25T22:28:42.307 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:949: test_activate_osd_skip_benchmark: local 'max_iops_hdd_after_boot={"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-25T22:28:42.308 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: get_asok_path osd.0 2026-02-25T22:28:42.308 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:28:42.309 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:28:42.309 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:42.310 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:42.310 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:42.310 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-osd.0.asok 2026-02-25T22:28:42.310 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: CEPH_ARGS= 2026-02-25T22:28:42.310 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: ceph --format=json daemon /tmp/ceph-asok.55572/ceph-osd.0.asok config get osd_mclock_max_capacity_iops_ssd 2026-02-25T22:28:42.370 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:951: test_activate_osd_skip_benchmark: local 'max_iops_ssd_after_boot={"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-25T22:28:42.370 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:953: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' = '{"osd_mclock_max_capacity_iops_hdd":"315.000000"}' 2026-02-25T22:28:42.370 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:954: test_activate_osd_skip_benchmark: test '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' = '{"osd_mclock_max_capacity_iops_ssd":"21500.000000"}' 2026-02-25T22:28:42.370 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:956: test_activate_osd_skip_benchmark: teardown td/ceph-helpers 2026-02-25T22:28:42.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:42.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:42.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:42.371 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:42.371 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:42.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:42.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:42.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:42.478 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:42.478 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:42.479 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:42.479 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:42.480 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:42.480 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:42.481 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:42.482 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:42.482 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:42.482 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:42.483 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:42.483 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:42.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:42.485 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:42.489 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:42.489 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:42.489 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:42.489 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:42.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:42.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:42.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:28:42.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_destroy_osd td/ceph-helpers 2026-02-25T22:28:42.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:787: test_destroy_osd: local dir=td/ceph-helpers 2026-02-25T22:28:42.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:789: test_destroy_osd: setup td/ceph-helpers 2026-02-25T22:28:42.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:28:42.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:28:42.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:42.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:42.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:42.493 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:42.493 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:42.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:42.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:42.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:42.494 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:42.494 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:42.495 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:42.495 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:42.496 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:42.496 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:42.497 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:42.498 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:42.498 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:42.498 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:42.499 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:42.499 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:42.500 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:42.500 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:42.502 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:42.502 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:42.502 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:42.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:42.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:42.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:42.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:28:42.504 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:28:42.504 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:42.505 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:42.505 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:28:42.506 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:28:42.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:28:42.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:28:42.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:28:42.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:790: test_destroy_osd: run_mon td/ceph-helpers a 2026-02-25T22:28:42.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:28:42.508 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:28:42.508 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:28:42.508 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:28:42.508 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:28:42.508 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:28:42.527 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:28:42.528 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:42.528 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:42.528 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:42.528 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:42.528 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:42.528 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:42.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:28:42.558 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:28:42.559 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:28:42.559 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:42.559 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:42.559 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:28:42.560 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:28:42.560 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:42.560 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:42.560 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:42.560 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:42.561 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:42.561 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:42.561 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:42.561 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:42.561 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:28:42.614 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:28:42.614 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:42.615 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:42.615 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:28:42.616 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:28:42.616 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:42.616 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:42.616 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:42.616 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:42.616 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:42.616 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:42.616 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:42.617 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:42.617 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:28:42.669 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:791: test_destroy_osd: run_mgr td/ceph-helpers x 2026-02-25T22:28:42.669 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:28:42.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:28:42.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:28:42.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:28:42.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:28:42.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:28:42.777 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:28:42.778 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:42.778 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:42.778 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:42.778 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:42.779 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:42.779 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:42.779 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:42.779 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:42.801 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:792: test_destroy_osd: run_osd td/ceph-helpers 0 2026-02-25T22:28:42.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:28:42.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:42.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:28:42.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:42.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:28:42.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:28:42.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:42.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:42.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:42.802 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:28:42.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:28:42.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:42.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:42.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:28:42.803 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:42.803 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:42.803 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:42.803 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:42.803 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:42.803 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:42.803 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:42.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:42.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:42.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:42.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:42.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:28:42.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:28:42.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:42.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:42.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:42.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:42.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:42.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:42.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:28:42.805 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:42.805 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d071004c-f7b4-4d72-b33c-074bcc92fd63 2026-02-25T22:28:42.805 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 d071004c-f7b4-4d72-b33c-074bcc92fd63 2026-02-25T22:28:42.806 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d071004c-f7b4-4d72-b33c-074bcc92fd63' 2026-02-25T22:28:42.806 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:42.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCad59pQXZwMBAA3vJq1oyH5sxcpMTOwaO72A== 2026-02-25T22:28:42.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCad59pQXZwMBAA3vJq1oyH5sxcpMTOwaO72A=="}' 2026-02-25T22:28:42.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d071004c-f7b4-4d72-b33c-074bcc92fd63 -i td/ceph-helpers/0/new.json 2026-02-25T22:28:42.929 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:42.937 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:28:42.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCad59pQXZwMBAA3vJq1oyH5sxcpMTOwaO72A== --osd-uuid d071004c-f7b4-4d72-b33c-074bcc92fd63 2026-02-25T22:28:42.954 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:42.946+0000 7f31a3670a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:42.956 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:42.950+0000 7f31a3670a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:42.957 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:42.950+0000 7f31a3670a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:42.957 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:42.950+0000 7f31a3670a40 -1 bdev(0x560df74c2400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:42.957 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:42.950+0000 7f31a3670a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:28:43.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:28:43.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:43.635 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:28:43.635 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:28:43.636 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:43.750 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:28:43.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:28:43.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:43.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:43.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:43.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:43.765 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:43.758+0000 7f6085ffda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:43.767 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:43.758+0000 7f6085ffda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:43.768 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:43.762+0000 7f6085ffda40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:43.864 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:28:43.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:43.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:43.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:43.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:43.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:43.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:43.865 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:43.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:43.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:43.977 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:44.192 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:44.186+0000 7f6085ffda40 -1 Falling back to public interface 2026-02-25T22:28:44.824 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:44.818+0000 7f6085ffda40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:28:44.979 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:44.979 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:44.979 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:44.979 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:28:44.980 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:44.980 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:45.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:46.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:46.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:46.143 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:28:46.144 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:46.144 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:46.144 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:46.352 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2538944397,v1:127.0.0.1:6801/2538944397] [v2:127.0.0.1:6802/2538944397,v1:127.0.0.1:6803/2538944397] exists,up d071004c-f7b4-4d72-b33c-074bcc92fd63 2026-02-25T22:28:46.353 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:46.353 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:46.354 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:46.354 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:793: test_destroy_osd: destroy_osd td/ceph-helpers 0 2026-02-25T22:28:46.354 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:775: destroy_osd: local dir=td/ceph-helpers 2026-02-25T22:28:46.354 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:776: destroy_osd: local id=0 2026-02-25T22:28:46.354 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:778: destroy_osd: ceph osd out osd.0 2026-02-25T22:28:46.618 INFO:tasks.workunit.client.0.trial188.stderr:osd.0 is already out. 2026-02-25T22:28:46.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:779: destroy_osd: kill_daemons td/ceph-helpers TERM osd.0 2026-02-25T22:28:46.630 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:46.630 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:46.631 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:46.631 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:46.631 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:46.939 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:46.939 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:780: destroy_osd: ceph osd down osd.0 2026-02-25T22:28:47.145 INFO:tasks.workunit.client.0.trial188.stderr:osd.0 is already down. 2026-02-25T22:28:47.154 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:781: destroy_osd: ceph osd purge osd.0 --yes-i-really-mean-it 2026-02-25T22:28:47.372 INFO:tasks.workunit.client.0.trial188.stderr:osd.0 does not exist 2026-02-25T22:28:47.382 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:782: destroy_osd: teardown td/ceph-helpers/0 2026-02-25T22:28:47.382 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers/0 2026-02-25T22:28:47.382 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:47.382 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers/0 KILL 2026-02-25T22:28:47.382 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:47.382 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:47.383 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:47.383 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:47.383 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:47.384 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:47.385 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:47.385 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:47.386 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:47.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:47.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:47.388 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:47.389 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:47.389 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:47.389 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:47.389 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:47.390 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:47.392 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:47.392 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers/0 2026-02-25T22:28:47.394 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:47.394 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:47.395 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:47.395 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:47.396 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:47.396 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:47.396 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:783: destroy_osd: rm -fr td/ceph-helpers/0 2026-02-25T22:28:47.397 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:794: test_destroy_osd: ceph osd dump 2026-02-25T22:28:47.397 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:794: test_destroy_osd: grep 'osd. ' 2026-02-25T22:28:47.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:795: test_destroy_osd: teardown td/ceph-helpers 2026-02-25T22:28:47.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:47.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:47.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:47.620 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:47.620 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:47.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:47.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:47.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:47.727 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:47.728 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:47.728 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:47.729 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:47.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:47.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:47.731 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:47.732 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:47.732 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:47.732 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:47.732 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:47.733 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:47.734 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:47.735 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:47.738 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:47.738 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:47.738 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:47.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:47.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_display_logs td/ceph-helpers 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2158: test_display_logs: local dir=td/ceph-helpers 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2160: test_display_logs: setup td/ceph-helpers 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:47.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:47.742 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:47.742 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:47.743 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:47.743 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:47.745 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:47.745 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:47.745 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:47.746 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:47.746 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:47.746 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:47.747 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:47.747 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:47.748 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:47.748 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:47.749 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:47.749 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:47.749 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:47.750 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:47.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:47.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:47.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:28:47.752 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:28:47.752 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:47.752 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:47.752 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:28:47.754 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:28:47.754 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:28:47.754 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:28:47.755 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:28:47.755 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2161: test_display_logs: run_mon td/ceph-helpers a 2026-02-25T22:28:47.755 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:28:47.755 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:28:47.755 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:28:47.755 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:28:47.755 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:28:47.755 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:28:47.776 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:28:47.777 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:47.777 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:47.777 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:47.777 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:47.777 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:47.778 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:47.778 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:28:47.807 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:28:47.808 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:28:47.808 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:47.809 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:47.809 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:28:47.810 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:28:47.810 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:47.810 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:47.810 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:47.810 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:47.810 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:47.810 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:47.811 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:47.811 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:47.811 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:28:47.868 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:28:47.868 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:47.868 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:47.868 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:28:47.868 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:28:47.868 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:47.868 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:47.868 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:47.869 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:47.869 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:47.869 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:47.869 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:47.869 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:47.869 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:28:47.925 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2162: test_display_logs: kill_daemons td/ceph-helpers 2026-02-25T22:28:47.926 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:47.926 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:47.926 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:47.926 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:47.926 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:48.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:48.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2163: test_display_logs: display_logs td/ceph-helpers 2026-02-25T22:28:48.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2148: display_logs: local dir=td/ceph-helpers 2026-02-25T22:28:48.033 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2150: display_logs: find td/ceph-helpers -maxdepth 1 -name '*.log' 2026-02-25T22:28:48.033 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: display_logs: read file 2026-02-25T22:28:48.034 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2152: display_logs: echo '======================= td/ceph-helpers/mon.a.log' 2026-02-25T22:28:48.034 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2153: display_logs: cat td/ceph-helpers/mon.a.log 2026-02-25T22:28:48.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2151: display_logs: read file 2026-02-25T22:28:48.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2164: test_display_logs: grep --quiet mon.a.log td/ceph-helpers/log.out 2026-02-25T22:28:48.036 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2165: test_display_logs: teardown td/ceph-helpers 2026-02-25T22:28:48.036 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:48.036 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:48.036 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:48.036 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:48.036 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:48.037 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:48.037 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:48.037 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:48.039 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:48.039 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:48.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:48.041 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:48.042 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:48.042 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:48.042 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:48.043 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.043 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:48.044 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:48.044 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.045 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:48.046 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:48.046 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:48.047 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:48.047 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.048 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.048 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:48.049 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:48.049 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:48.049 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:28:48.049 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_erasure_code_plugin_exists td/ceph-helpers 2026-02-25T22:28:48.049 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2129: test_erasure_code_plugin_exists: local dir=td/ceph-helpers 2026-02-25T22:28:48.049 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2131: test_erasure_code_plugin_exists: setup td/ceph-helpers 2026-02-25T22:28:48.049 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:28:48.049 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:28:48.049 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:48.050 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:48.050 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:48.050 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:48.050 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:48.050 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:48.050 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:48.050 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:48.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:48.052 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:48.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:48.054 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:48.055 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:48.055 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:48.055 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:48.056 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.056 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:48.057 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:48.057 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.058 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:48.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:48.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:48.060 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:48.060 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.060 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.061 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:48.061 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:48.062 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:48.062 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:28:48.063 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:28:48.063 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.063 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.063 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:28:48.064 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:28:48.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:28:48.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:28:48.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:28:48.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2132: test_erasure_code_plugin_exists: run_mon td/ceph-helpers a 2026-02-25T22:28:48.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:28:48.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:28:48.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:28:48.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:28:48.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:28:48.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:28:48.086 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:28:48.086 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:48.086 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:48.086 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:48.086 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.087 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.087 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:48.087 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:28:48.115 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:28:48.116 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:28:48.116 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:48.116 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:48.116 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:28:48.116 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:28:48.116 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:48.117 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:48.117 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:48.117 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:48.117 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.117 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.117 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:48.117 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:48.118 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:28:48.173 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:28:48.173 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:48.173 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:48.173 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:28:48.173 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:28:48.174 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:48.174 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:48.174 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:48.174 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:48.174 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.174 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.174 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:48.175 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:48.175 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:28:48.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2133: test_erasure_code_plugin_exists: run_mgr td/ceph-helpers x 2026-02-25T22:28:48.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:28:48.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:28:48.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:28:48.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:28:48.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:28:48.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:28:48.340 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:28:48.340 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:48.340 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:48.340 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:48.340 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.340 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.341 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:48.341 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:48.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:48.361 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2134: test_erasure_code_plugin_exists: erasure_code_plugin_exists jerasure 2026-02-25T22:28:48.361 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2104: erasure_code_plugin_exists: local plugin=jerasure 2026-02-25T22:28:48.361 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2105: erasure_code_plugin_exists: local status 2026-02-25T22:28:48.361 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: erasure_code_plugin_exists: local grepstr 2026-02-25T22:28:48.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: erasure_code_plugin_exists: local s 2026-02-25T22:28:48.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: case `uname` in 2026-02-25T22:28:48.362 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: uname 2026-02-25T22:28:48.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: erasure_code_plugin_exists: grepstr='jerasure.*No such file' 2026-02-25T22:28:48.362 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=jerasure 2026-02-25T22:28:48.482 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: s= 2026-02-25T22:28:48.482 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: erasure_code_plugin_exists: local status=0 2026-02-25T22:28:48.482 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: erasure_code_plugin_exists: '[' 0 -eq 0 ']' 2026-02-25T22:28:48.482 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2116: erasure_code_plugin_exists: ceph osd erasure-code-profile rm TESTPROFILE 2026-02-25T22:28:48.639 INFO:tasks.workunit.client.0.trial188.stderr:erasure-code-profile TESTPROFILE does not exist 2026-02-25T22:28:48.647 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: erasure_code_plugin_exists: return 0 2026-02-25T22:28:48.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2135: test_erasure_code_plugin_exists: erasure_code_plugin_exists FAKE 2026-02-25T22:28:48.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2104: erasure_code_plugin_exists: local plugin=FAKE 2026-02-25T22:28:48.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2105: erasure_code_plugin_exists: local status 2026-02-25T22:28:48.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2106: erasure_code_plugin_exists: local grepstr 2026-02-25T22:28:48.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2107: erasure_code_plugin_exists: local s 2026-02-25T22:28:48.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: case `uname` in 2026-02-25T22:28:48.648 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2108: erasure_code_plugin_exists: uname 2026-02-25T22:28:48.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2110: erasure_code_plugin_exists: grepstr='FAKE.*No such file' 2026-02-25T22:28:48.649 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: ceph osd erasure-code-profile set TESTPROFILE plugin=FAKE 2026-02-25T22:28:48.745 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2113: erasure_code_plugin_exists: s='Error EIO: load dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so): /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory' 2026-02-25T22:28:48.745 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2114: erasure_code_plugin_exists: local status=5 2026-02-25T22:28:48.745 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2115: erasure_code_plugin_exists: '[' 5 -eq 0 ']' 2026-02-25T22:28:48.745 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: erasure_code_plugin_exists: echo Error EIO: load 'dlopen(/usr/lib/ceph/erasure-code/libec_FAKE.so):' /usr/lib/ceph/erasure-code/libec_FAKE.so: cannot open shared object file: No such file or directory 2026-02-25T22:28:48.746 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2117: erasure_code_plugin_exists: grep --quiet 'FAKE.*No such file' 2026-02-25T22:28:48.747 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2119: erasure_code_plugin_exists: status=1 2026-02-25T22:28:48.747 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2125: erasure_code_plugin_exists: return 1 2026-02-25T22:28:48.748 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2136: test_erasure_code_plugin_exists: teardown td/ceph-helpers 2026-02-25T22:28:48.748 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:48.748 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:48.748 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:48.748 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:48.748 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:48.748 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:48.748 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:48.748 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:48.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:48.854 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:48.855 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:48.855 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:48.856 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:48.856 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:48.856 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:48.857 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.858 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:48.858 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:48.858 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.859 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:48.860 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:48.860 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:48.862 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:48.862 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.862 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.862 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:48.864 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:48.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:48.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:28:48.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_expect_failure td/ceph-helpers 2026-02-25T22:28:48.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2037: test_expect_failure: local dir=td/ceph-helpers 2026-02-25T22:28:48.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2039: test_expect_failure: setup td/ceph-helpers 2026-02-25T22:28:48.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:28:48.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:28:48.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:48.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:48.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:48.865 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:48.865 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:48.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:48.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:48.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:48.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:48.867 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:48.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:48.868 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:48.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:48.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:48.869 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:48.870 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.870 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:48.871 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:48.871 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.871 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:48.872 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:48.872 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:48.874 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:48.874 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.874 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.874 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:48.875 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:48.875 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:48.875 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:28:48.876 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:28:48.876 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.877 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.877 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:28:48.878 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:28:48.879 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:28:48.879 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:28:48.879 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:28:48.879 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2040: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 1' 2026-02-25T22:28:48.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/ceph-helpers 2026-02-25T22:28:48.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:28:48.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local expected=FAIL 2026-02-25T22:28:48.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:28:48.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:28:48.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: bash -c 'echo FAIL ; exit 1' 2026-02-25T22:28:48.881 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-25T22:28:48.881 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-25T22:28:48.881 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-25T22:28:48.882 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2032: expect_failure: return 0 2026-02-25T22:28:48.882 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2042: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo FAIL ; exit 0' 2026-02-25T22:28:48.882 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/ceph-helpers 2026-02-25T22:28:48.882 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:28:48.882 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local expected=FAIL 2026-02-25T22:28:48.882 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:28:48.882 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:28:48.882 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: bash -c 'echo FAIL ; exit 0' 2026-02-25T22:28:48.883 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2023: expect_failure: success=true 2026-02-25T22:28:48.883 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: true 2026-02-25T22:28:48.883 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: expect_failure: cat td/ceph-helpers/out 2026-02-25T22:28:48.884 INFO:tasks.workunit.client.0.trial188.stderr:FAIL 2026-02-25T22:28:48.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: expect_failure: return 1 2026-02-25T22:28:48.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2043: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-25T22:28:48.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2045: test_expect_failure: expect_failure td/ceph-helpers FAIL bash -c 'echo UNEXPECTED ; exit 1' 2026-02-25T22:28:48.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2016: expect_failure: local dir=td/ceph-helpers 2026-02-25T22:28:48.886 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2017: expect_failure: shift 2026-02-25T22:28:48.886 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2018: expect_failure: local expected=FAIL 2026-02-25T22:28:48.886 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2019: expect_failure: shift 2026-02-25T22:28:48.886 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2020: expect_failure: local success 2026-02-25T22:28:48.886 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2022: expect_failure: bash -c 'echo UNEXPECTED ; exit 1' 2026-02-25T22:28:48.887 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2025: expect_failure: success=false 2026-02-25T22:28:48.887 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: false 2026-02-25T22:28:48.887 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2028: expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-25T22:28:48.888 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2029: expect_failure: cat td/ceph-helpers/out 2026-02-25T22:28:48.889 INFO:tasks.workunit.client.0.trial188.stderr:UNEXPECTED 2026-02-25T22:28:48.890 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2030: expect_failure: return 1 2026-02-25T22:28:48.890 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2046: test_expect_failure: grep --quiet FAIL td/ceph-helpers/out 2026-02-25T22:28:48.891 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2047: test_expect_failure: teardown td/ceph-helpers 2026-02-25T22:28:48.891 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:48.891 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:48.891 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:48.891 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:48.891 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:48.892 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:48.892 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:48.892 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:48.894 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:48.894 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:48.895 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:48.895 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:48.896 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:48.896 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:48.896 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:48.897 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.898 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:48.898 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:48.898 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.899 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:48.900 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:48.900 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:48.901 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:48.901 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.902 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.902 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:48.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:48.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:48.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:28:48.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_file_with_random_data td/ceph-helpers 2026-02-25T22:28:48.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2350: test_file_with_random_data: local dir=td/ceph-helpers 2026-02-25T22:28:48.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2352: test_file_with_random_data: setup td/ceph-helpers 2026-02-25T22:28:48.904 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:28:48.905 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:28:48.905 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:48.905 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:48.905 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:48.905 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:48.905 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:48.905 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:48.905 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:48.905 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:48.906 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:48.906 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:48.907 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:48.907 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:48.909 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:48.909 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:48.909 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:48.910 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.910 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:48.911 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:48.911 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.912 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:48.913 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:48.913 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:48.914 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:48.914 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.914 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.914 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:48.916 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:48.916 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:48.916 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:28:48.917 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:28:48.917 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.917 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.917 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:28:48.918 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:28:48.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:28:48.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:28:48.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:28:48.920 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: test_file_with_random_data: file_with_random_data 4000 2026-02-25T22:28:48.920 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2343: file_with_random_data: local size_bytes=4000 2026-02-25T22:28:48.920 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: file_with_random_data: mktemp 2026-02-25T22:28:48.921 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2344: file_with_random_data: local file=/tmp/tmp.bRp2zaxZBJ 2026-02-25T22:28:48.921 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2345: file_with_random_data: dd if=/dev/urandom of=/tmp/tmp.bRp2zaxZBJ bs=4000 count=1 2026-02-25T22:28:48.922 INFO:tasks.workunit.client.0.trial188.stderr:1+0 records in 2026-02-25T22:28:48.922 INFO:tasks.workunit.client.0.trial188.stderr:1+0 records out 2026-02-25T22:28:48.922 INFO:tasks.workunit.client.0.trial188.stderr:4000 bytes (4.0 kB, 3.9 KiB) copied, 5.5514e-05 s, 72.1 MB/s 2026-02-25T22:28:48.922 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2346: file_with_random_data: printf %s /tmp/tmp.bRp2zaxZBJ 2026-02-25T22:28:48.922 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2353: test_file_with_random_data: local file=/tmp/tmp.bRp2zaxZBJ 2026-02-25T22:28:48.922 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2354: test_file_with_random_data: test -f /tmp/tmp.bRp2zaxZBJ 2026-02-25T22:28:48.923 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: test_file_with_random_data: stat -c %s /tmp/tmp.bRp2zaxZBJ 2026-02-25T22:28:48.923 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2355: test_file_with_random_data: test 4000 = 4000 2026-02-25T22:28:48.923 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2356: test_file_with_random_data: rm /tmp/tmp.bRp2zaxZBJ 2026-02-25T22:28:48.925 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2357: test_file_with_random_data: teardown td/ceph-helpers 2026-02-25T22:28:48.925 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:48.925 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:48.925 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:48.926 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:48.926 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:48.926 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:48.926 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:48.926 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:48.928 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:48.928 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:48.928 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:48.929 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:48.930 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:48.930 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:48.930 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:48.931 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.931 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:48.932 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:48.932 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.933 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:48.934 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:48.934 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:48.935 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:48.935 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.935 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.935 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:48.937 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:48.937 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:48.937 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:28:48.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_flush_pg_stats td/ceph-helpers 2026-02-25T22:28:48.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2280: test_flush_pg_stats: local dir=td/ceph-helpers 2026-02-25T22:28:48.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2282: test_flush_pg_stats: setup td/ceph-helpers 2026-02-25T22:28:48.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:28:48.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:28:48.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:48.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:48.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:48.938 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:48.938 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:48.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:48.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:48.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:48.939 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:48.939 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:48.940 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:48.940 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:48.941 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:48.941 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:48.942 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:48.943 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.943 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:48.943 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:48.943 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:48.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:48.945 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:48.945 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:48.946 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:48.946 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.947 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.947 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:48.948 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:48.948 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:48.948 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:28:48.949 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:28:48.949 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.949 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:28:48.950 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:28:48.951 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:28:48.951 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:28:48.951 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:28:48.951 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2283: test_flush_pg_stats: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:28:48.951 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:28:48.952 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:28:48.952 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:28:48.952 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:28:48.952 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:28:48.952 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:28:48.972 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:28:48.972 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:48.972 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:48.972 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:48.972 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:48.973 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:48.973 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:48.973 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:28:49.002 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:28:49.003 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:28:49.003 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:49.003 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:49.003 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:28:49.004 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:28:49.004 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:49.005 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:49.005 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:49.005 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:49.005 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:49.005 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:49.005 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:49.005 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:49.005 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:28:49.059 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:28:49.059 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:49.059 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:49.059 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:28:49.060 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:28:49.061 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:49.061 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:49.061 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:49.061 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:49.061 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:49.061 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:49.061 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:49.061 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:49.061 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:28:49.114 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2284: test_flush_pg_stats: run_mgr td/ceph-helpers x 2026-02-25T22:28:49.114 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:28:49.114 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:28:49.114 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:28:49.115 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:28:49.115 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:28:49.115 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:28:49.229 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:28:49.230 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:49.230 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:49.230 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:49.230 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:49.231 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:49.231 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:49.231 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:49.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:49.251 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2285: test_flush_pg_stats: run_osd td/ceph-helpers 0 2026-02-25T22:28:49.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:28:49.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:49.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:28:49.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:49.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:28:49.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:28:49.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:49.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:49.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:49.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:28:49.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:28:49.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:49.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:49.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:28:49.253 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:49.253 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:49.253 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:49.253 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:49.253 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:49.253 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:49.253 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:49.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:49.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:49.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:49.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:49.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:28:49.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:28:49.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:49.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:49.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:49.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:49.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:49.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:28:49.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:28:49.255 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:49.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=92f290e8-a14c-4266-bb3e-1953739ff10a 2026-02-25T22:28:49.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 92f290e8-a14c-4266-bb3e-1953739ff10a' 2026-02-25T22:28:49.255 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 92f290e8-a14c-4266-bb3e-1953739ff10a 2026-02-25T22:28:49.256 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:49.266 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQChd59ppei1DxAATTI9I6X/3nHmx9xA7GakRQ== 2026-02-25T22:28:49.266 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQChd59ppei1DxAATTI9I6X/3nHmx9xA7GakRQ=="}' 2026-02-25T22:28:49.266 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 92f290e8-a14c-4266-bb3e-1953739ff10a -i td/ceph-helpers/0/new.json 2026-02-25T22:28:49.380 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:49.388 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:28:49.389 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQChd59ppei1DxAATTI9I6X/3nHmx9xA7GakRQ== --osd-uuid 92f290e8-a14c-4266-bb3e-1953739ff10a 2026-02-25T22:28:49.405 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:49.398+0000 7fc740ab1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:49.407 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:49.398+0000 7fc740ab1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:49.408 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:49.402+0000 7fc740ab1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:49.408 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:49.402+0000 7fc740ab1a40 -1 bdev(0x556a5d002400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:49.408 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:49.402+0000 7fc740ab1a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:28:50.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:28:50.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:50.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:28:50.143 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:28:50.144 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:50.263 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:28:50.264 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:28:50.264 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:28:50.264 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:50.264 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:50.264 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:50.286 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:50.278+0000 7f9b04c24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:50.288 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:50.282+0000 7f9b04c24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:50.290 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:50.282+0000 7f9b04c24a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:50.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:28:50.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:50.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:50.374 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:50.374 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:50.374 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:50.374 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:50.374 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:50.375 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:50.375 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:50.483 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:50.680 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:50.674+0000 7f9b04c24a40 -1 Falling back to public interface 2026-02-25T22:28:51.312 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:51.306+0000 7f9b04c24a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:28:51.484 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:28:51.485 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:51.485 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:51.485 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:51.485 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:51.485 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:51.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:52.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:52.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:52.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:28:52.602 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:28:52.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:52.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:52.813 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3225619268,v1:127.0.0.1:6801/3225619268] [v2:127.0.0.1:6802/3225619268,v1:127.0.0.1:6803/3225619268] exists,up 92f290e8-a14c-4266-bb3e-1953739ff10a 2026-02-25T22:28:52.813 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:28:52.813 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:28:52.813 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:28:52.813 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2286: test_flush_pg_stats: create_rbd_pool 2026-02-25T22:28:52.813 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:28:53.015 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:28:53.025 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:28:53.025 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:28:53.271 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:28:53.280 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:28:54.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:28:54.557 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2287: test_flush_pg_stats: rados -p rbd put obj /etc/group 2026-02-25T22:28:54.574 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2288: test_flush_pg_stats: flush_pg_stats 2026-02-25T22:28:54.574 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:28:54.575 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:28:54.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:28:54.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:28:54.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:28:54.784 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:28:54.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:28:54.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:28:54.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:28:54.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:28:54.867 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:28:54.867 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:28:54.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:28:54.868 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:28:54.869 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:28:54.870 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:28:54.870 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:28:54.871 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:28:54.871 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:28:55.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:28:55.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:28:56.096 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:28:56.096 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:28:56.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:28:56.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2289: test_flush_pg_stats: local 'jq_filter=.pools | .[] | select(.name == "rbd") | .stats' 2026-02-25T22:28:56.318 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: test_flush_pg_stats: ceph df detail --format=json 2026-02-25T22:28:56.318 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored' 2026-02-25T22:28:56.605 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2290: test_flush_pg_stats: stored=8398 2026-02-25T22:28:56.606 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: test_flush_pg_stats: ceph df detail --format=json 2026-02-25T22:28:56.606 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: test_flush_pg_stats: jq '.pools | .[] | select(.name == "rbd") | .stats.stored_raw' 2026-02-25T22:28:56.898 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2291: test_flush_pg_stats: stored_raw=8398 2026-02-25T22:28:56.898 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2292: test_flush_pg_stats: test 8398 -gt 0 2026-02-25T22:28:56.898 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2293: test_flush_pg_stats: test 8398 == 8398 2026-02-25T22:28:56.898 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2294: test_flush_pg_stats: teardown td/ceph-helpers 2026-02-25T22:28:56.898 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:56.898 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:56.898 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:56.898 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:56.899 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:56.899 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:56.899 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:56.899 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:57.005 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:57.006 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:57.006 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:57.006 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:57.007 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:57.007 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:57.008 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:57.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:57.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:57.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:57.009 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:57.010 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:57.011 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:57.012 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:57.017 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:57.017 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:57.017 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:57.017 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:57.020 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:57.020 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:57.020 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:28:57.020 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_config td/ceph-helpers 2026-02-25T22:28:57.020 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1127: test_get_config: local dir=td/ceph-helpers 2026-02-25T22:28:57.020 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1130: test_get_config: setup td/ceph-helpers 2026-02-25T22:28:57.020 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:28:57.020 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:28:57.020 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:28:57.020 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:28:57.020 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:28:57.020 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:28:57.021 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:28:57.021 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:28:57.021 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:28:57.021 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:28:57.021 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:28:57.022 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:28:57.023 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:28:57.023 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:28:57.024 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:28:57.024 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:28:57.024 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:28:57.025 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:57.026 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:28:57.026 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:28:57.026 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:28:57.027 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:28:57.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:28:57.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:28:57.030 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:28:57.030 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:57.030 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:57.030 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:28:57.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:28:57.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:28:57.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:28:57.032 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:28:57.032 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:57.032 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:57.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:28:57.034 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:28:57.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:28:57.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:28:57.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:28:57.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1131: test_get_config: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:28:57.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:28:57.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:28:57.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:28:57.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:28:57.036 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:28:57.036 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:28:57.056 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:28:57.056 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:57.056 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:57.056 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:57.056 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:57.056 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:57.057 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:57.057 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:28:57.088 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:28:57.089 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:28:57.089 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:57.089 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:57.089 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:28:57.090 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:28:57.090 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:57.090 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:57.090 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:57.090 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:57.091 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:57.091 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:57.091 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:57.091 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:57.091 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:28:57.146 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:28:57.146 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:57.146 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:57.146 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:28:57.146 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:28:57.147 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:57.147 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:57.147 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:57.148 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:57.148 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:57.148 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:57.148 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:57.148 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:57.148 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:28:57.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: test_get_config: get_config mon a osd_pool_default_size 2026-02-25T22:28:57.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:28:57.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:28:57.203 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_pool_default_size 2026-02-25T22:28:57.203 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_pool_default_size 2026-02-25T22:28:57.204 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:28:57.204 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:28:57.204 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:28:57.204 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:28:57.204 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:57.204 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:57.204 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:28:57.204 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:28:57.204 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get osd_pool_default_size 2026-02-25T22:28:57.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1132: test_get_config: test 1 = 1 2026-02-25T22:28:57.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1133: test_get_config: run_mgr td/ceph-helpers x 2026-02-25T22:28:57.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:28:57.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:28:57.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:28:57.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:28:57.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:28:57.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:28:57.375 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:28:57.375 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:57.376 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:57.376 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:57.376 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:57.376 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:57.377 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:57.377 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:57.378 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:28:57.398 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1134: test_get_config: run_osd td/ceph-helpers 0 --osd_max_scrubs=3 2026-02-25T22:28:57.398 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:28:57.398 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:28:57.398 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:28:57.398 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:28:57.398 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:28:57.399 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:28:57.399 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:28:57.399 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:28:57.399 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:28:57.399 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:28:57.399 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:28:57.399 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:28:57.399 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:28:57.399 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:28:57.399 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:28:57.399 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:28:57.399 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:28:57.400 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:28:57.400 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:28:57.400 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:28:57.400 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:57.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:28:57.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:28:57.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:28:57.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:28:57.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:28:57.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:28:57.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:28:57.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:28:57.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:28:57.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:28:57.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:28:57.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_max_scrubs=3 2026-02-25T22:28:57.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:28:57.401 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:28:57.402 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=fd955f78-0c0e-4d40-88b9-3975799becf2 2026-02-25T22:28:57.402 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 fd955f78-0c0e-4d40-88b9-3975799becf2' 2026-02-25T22:28:57.402 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 fd955f78-0c0e-4d40-88b9-3975799becf2 2026-02-25T22:28:57.402 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:28:57.412 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCpd59pN0xmGBAA8/DM1x/m4dQr/NtHQcJGDQ== 2026-02-25T22:28:57.412 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCpd59pN0xmGBAA8/DM1x/m4dQr/NtHQcJGDQ=="}' 2026-02-25T22:28:57.412 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new fd955f78-0c0e-4d40-88b9-3975799becf2 -i td/ceph-helpers/0/new.json 2026-02-25T22:28:57.526 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:57.534 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:28:57.535 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 --mkfs --key AQCpd59pN0xmGBAA8/DM1x/m4dQr/NtHQcJGDQ== --osd-uuid fd955f78-0c0e-4d40-88b9-3975799becf2 2026-02-25T22:28:57.552 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:57.546+0000 7f9b2c0ffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:57.553 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:57.546+0000 7f9b2c0ffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:57.554 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:57.546+0000 7f9b2c0ffa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:57.555 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:57.546+0000 7f9b2c0ffa40 -1 bdev(0x56394718c400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:28:57.555 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:57.546+0000 7f9b2c0ffa40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:28:58.278 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:28:58.278 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:28:58.279 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:28:58.280 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:28:58.280 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:28:58.401 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:28:58.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:28:58.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_max_scrubs=3 2026-02-25T22:28:58.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:28:58.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:28:58.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:28:58.425 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:58.418+0000 7fa8fefd0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:58.429 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:58.422+0000 7fa8fefd0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:58.430 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:58.422+0000 7fa8fefd0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:28:58.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:28:58.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:28:58.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:28:58.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:28:58.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:28:58.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:58.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:28:58.511 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:28:58.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:58.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:58.626 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:58.980 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:58.974+0000 7fa8fefd0a40 -1 Falling back to public interface 2026-02-25T22:28:59.627 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:28:59.627 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:28:59.628 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:28:59.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:28:59.632 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:28:59.632 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:28:59.743 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:28:59.808 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:28:59.802+0000 7fa8fefd0a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:29:00.744 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:00.744 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:00.745 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:29:00.745 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:00.745 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:00.745 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:00.953 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/478933904,v1:127.0.0.1:6801/478933904] [v2:127.0.0.1:6802/478933904,v1:127.0.0.1:6803/478933904] exists,up fd955f78-0c0e-4d40-88b9-3975799becf2 2026-02-25T22:29:00.953 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:00.954 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:00.954 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:00.954 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: test_get_config: get_config osd 0 osd_max_scrubs 2026-02-25T22:29:00.954 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2026-02-25T22:29:00.954 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=0 2026-02-25T22:29:00.954 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_max_scrubs 2026-02-25T22:29:00.954 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_max_scrubs 2026-02-25T22:29:00.955 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.0 2026-02-25T22:29:00.955 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:29:00.955 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:29:00.955 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:00.955 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:00.955 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:00.956 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-osd.0.asok 2026-02-25T22:29:00.956 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:00.956 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-osd.0.asok config get osd_max_scrubs 2026-02-25T22:29:01.021 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1135: test_get_config: test 3 = 3 2026-02-25T22:29:01.021 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1136: test_get_config: teardown td/ceph-helpers 2026-02-25T22:29:01.021 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:01.021 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:01.021 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:01.021 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:01.021 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:01.021 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:01.021 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:01.021 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:01.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:01.130 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:01.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:01.131 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:01.132 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:01.132 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:01.132 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:01.134 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:01.134 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:01.134 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:01.134 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:01.135 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:01.136 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:01.136 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:01.141 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:01.141 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.141 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:01.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:01.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:01.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:29:01.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_is_making_recovery_progress td/ceph-helpers 2026-02-25T22:29:01.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1335: test_get_is_making_recovery_progress: local dir=td/ceph-helpers 2026-02-25T22:29:01.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1337: test_get_is_making_recovery_progress: setup td/ceph-helpers 2026-02-25T22:29:01.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:29:01.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:29:01.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:01.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:01.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:01.143 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:01.143 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:01.144 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:01.144 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:01.144 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:01.146 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:01.146 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:01.147 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:01.147 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:01.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:01.148 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:01.149 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:01.150 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:01.150 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:01.150 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:01.150 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:01.151 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:01.152 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:01.152 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:01.154 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:01.154 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.154 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.154 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:01.155 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:01.155 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:01.155 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:29:01.156 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:29:01.157 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.157 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.157 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:29:01.158 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:29:01.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:29:01.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:29:01.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:29:01.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1338: test_get_is_making_recovery_progress: run_mon td/ceph-helpers a 2026-02-25T22:29:01.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:29:01.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:29:01.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:29:01.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:29:01.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:29:01.159 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:29:01.181 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:29:01.181 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:01.181 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:01.181 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:01.181 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.181 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.181 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:01.182 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:29:01.209 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:29:01.210 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:29:01.210 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:01.210 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:01.210 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:29:01.210 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:29:01.211 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:01.211 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:01.211 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:01.211 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:01.211 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.211 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.211 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:01.212 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:01.212 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:29:01.266 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:29:01.266 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:01.266 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:01.266 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:29:01.266 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:29:01.267 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:01.267 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:01.267 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:01.267 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:01.267 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.267 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.267 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:01.268 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:01.268 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:29:01.321 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1339: test_get_is_making_recovery_progress: run_mgr td/ceph-helpers x 2026-02-25T22:29:01.321 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:29:01.321 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:29:01.321 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:29:01.321 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:29:01.321 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:29:01.321 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:29:01.431 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:29:01.431 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:01.431 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:01.432 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:01.432 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.432 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.432 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:01.433 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:01.434 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:01.452 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1340: test_get_is_making_recovery_progress: get_is_making_recovery_progress 2026-02-25T22:29:01.452 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1325: get_is_making_recovery_progress: local recovery_progress 2026-02-25T22:29:01.452 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1326: get_is_making_recovery_progress: recovery_progress+='.recovering_keys_per_sec + ' 2026-02-25T22:29:01.452 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1327: get_is_making_recovery_progress: recovery_progress+='.recovering_bytes_per_sec + ' 2026-02-25T22:29:01.452 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1328: get_is_making_recovery_progress: recovery_progress+=.recovering_objects_per_sec 2026-02-25T22:29:01.452 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: ceph --format json status 2026-02-25T22:29:01.452 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: jq -r '.pgmap | .recovering_keys_per_sec + .recovering_bytes_per_sec + .recovering_objects_per_sec' 2026-02-25T22:29:01.571 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1330: get_is_making_recovery_progress: local progress=null 2026-02-25T22:29:01.571 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1331: get_is_making_recovery_progress: test null '!=' null 2026-02-25T22:29:01.571 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1341: test_get_is_making_recovery_progress: teardown td/ceph-helpers 2026-02-25T22:29:01.571 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:01.571 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:01.571 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:01.571 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:01.571 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:01.571 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:01.571 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:01.571 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:01.679 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:01.679 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:01.679 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:01.680 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:01.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:01.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:01.681 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:01.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:01.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:01.683 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:01.683 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:01.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:01.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:01.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:01.686 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:01.687 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.687 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.687 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:01.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:01.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:01.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:29:01.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_last_scrub_stamp td/ceph-helpers 2026-02-25T22:29:01.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1524: test_get_last_scrub_stamp: local dir=td/ceph-helpers 2026-02-25T22:29:01.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1526: test_get_last_scrub_stamp: setup td/ceph-helpers 2026-02-25T22:29:01.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:29:01.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:29:01.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:01.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:01.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:01.689 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:01.689 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:01.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:01.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:01.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:01.691 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:01.692 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:01.692 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:01.693 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:01.693 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:01.693 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:01.694 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:01.694 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:01.694 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:01.695 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:01.695 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:01.696 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:01.697 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:01.697 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:01.698 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:01.698 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.698 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.698 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:01.699 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:01.699 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:01.699 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:29:01.700 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:29:01.700 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.700 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.701 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:29:01.702 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:29:01.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:29:01.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:29:01.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:29:01.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1527: test_get_last_scrub_stamp: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:01.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:29:01.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:29:01.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:29:01.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:29:01.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:29:01.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:01.724 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:29:01.724 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:01.724 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:01.725 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:01.725 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.725 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.725 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:01.725 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:01.754 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:29:01.755 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:29:01.755 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:01.755 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:01.755 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:29:01.755 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:29:01.756 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:01.756 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:01.756 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:01.756 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:01.756 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.756 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.756 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:01.757 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:01.757 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:29:01.812 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:29:01.812 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:01.813 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:01.813 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:29:01.813 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:29:01.813 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:01.813 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:01.813 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:01.814 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:01.814 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.814 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.814 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:01.814 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:01.814 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:29:01.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1528: test_get_last_scrub_stamp: run_mgr td/ceph-helpers x 2026-02-25T22:29:01.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:29:01.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:29:01.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:29:01.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:29:01.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:29:01.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:29:01.980 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:29:01.980 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:01.980 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:01.981 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:01.981 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:01.981 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:01.981 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:01.981 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:01.982 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:02.001 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1529: test_get_last_scrub_stamp: run_osd td/ceph-helpers 0 2026-02-25T22:29:02.001 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:29:02.001 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:02.001 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:29:02.001 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:02.001 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:02.002 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:02.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:02.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:02.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:02.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:02.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:29:02.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:29:02.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:02.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:02.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:02.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:02.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:02.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:02.003 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:29:02.004 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:02.005 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f23cbc53-d6b5-47cd-8113-0c744ac314d5 2026-02-25T22:29:02.005 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 f23cbc53-d6b5-47cd-8113-0c744ac314d5 2026-02-25T22:29:02.005 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 f23cbc53-d6b5-47cd-8113-0c744ac314d5' 2026-02-25T22:29:02.005 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:02.017 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCud59p6oXiABAAuK7G7IIe7wsbWlzAF5qZPQ== 2026-02-25T22:29:02.017 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCud59p6oXiABAAuK7G7IIe7wsbWlzAF5qZPQ=="}' 2026-02-25T22:29:02.017 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f23cbc53-d6b5-47cd-8113-0c744ac314d5 -i td/ceph-helpers/0/new.json 2026-02-25T22:29:02.133 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:02.141 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:29:02.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCud59p6oXiABAAuK7G7IIe7wsbWlzAF5qZPQ== --osd-uuid f23cbc53-d6b5-47cd-8113-0c744ac314d5 2026-02-25T22:29:02.158 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:02.150+0000 7f315491ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:02.159 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:02.154+0000 7f315491ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:02.160 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:02.154+0000 7f315491ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:02.160 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:02.154+0000 7f315491ea40 -1 bdev(0x562cb0e3e400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:02.160 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:02.154+0000 7f315491ea40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:29:02.883 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:29:02.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:02.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:29:02.885 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:29:02.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:03.008 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:29:03.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:29:03.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:03.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:03.010 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:03.010 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:03.032 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:03.026+0000 7f360fe04a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:03.034 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:03.026+0000 7f360fe04a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:03.036 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:03.030+0000 7f360fe04a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:03.116 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:29:03.116 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:03.116 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:29:03.116 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:03.116 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:03.116 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:03.117 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:03.117 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:03.117 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:03.117 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:03.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:03.453 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:03.446+0000 7f360fe04a40 -1 Falling back to public interface 2026-02-25T22:29:04.088 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:04.082+0000 7f360fe04a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:29:04.234 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:04.234 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:04.234 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:29:04.235 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:04.238 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:04.238 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:04.389 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:05.391 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:05.391 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:05.391 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:29:05.392 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:05.392 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:05.392 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:05.604 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2324668027,v1:127.0.0.1:6801/2324668027] [v2:127.0.0.1:6802/2324668027,v1:127.0.0.1:6803/2324668027] exists,up f23cbc53-d6b5-47cd-8113-0c744ac314d5 2026-02-25T22:29:05.605 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:05.605 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:05.605 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:05.605 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1530: test_get_last_scrub_stamp: create_rbd_pool 2026-02-25T22:29:05.605 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:29:05.812 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:29:05.822 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:29:05.822 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:29:06.067 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:29:06.078 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:29:07.080 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:29:07.365 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1531: test_get_last_scrub_stamp: wait_for_clean 2026-02-25T22:29:07.365 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:29:07.366 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:29:07.366 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:29:07.366 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:29:07.367 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:29:07.367 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:29:07.367 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:29:07.367 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:29:07.367 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:29:07.471 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:29:07.471 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:29:07.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:29:07.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:29:07.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:29:07.472 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:29:07.695 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:29:07.696 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:29:07.696 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:07.696 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:29:07.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:29:07.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:29:07.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:29:07.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:07.777 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:29:07.777 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:07.778 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:29:07.779 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:29:07.779 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:07.780 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:29:07.780 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:29:07.780 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:29:07.781 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:07.997 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:29:07.997 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:29:08.999 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:29:08.999 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:09.217 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:29:09.218 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:29:09.218 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:09.218 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:09.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:29:09.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:29:09.510 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:29:09.511 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:29:09.511 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:29:09.511 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:29:09.511 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:29:09.511 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:29:09.722 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:29:09.722 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:29:09.723 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:09.723 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:10.012 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:29:10.012 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:29:10.012 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:29:10.013 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1532: test_get_last_scrub_stamp: get_last_scrub_stamp 1.0 2026-02-25T22:29:10.013 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:29:10.013 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:29:10.013 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:29:10.013 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:29:10.235 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1532: test_get_last_scrub_stamp: stamp=2026-02-25T22:29:06.009662+0000 2026-02-25T22:29:10.235 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1533: test_get_last_scrub_stamp: test -n 2026-02-25T22:29:06.009662+0000 2026-02-25T22:29:10.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1534: test_get_last_scrub_stamp: teardown td/ceph-helpers 2026-02-25T22:29:10.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:10.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:10.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:10.236 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:10.236 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:10.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:10.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:10.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:10.343 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:10.344 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:10.345 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:10.345 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:10.346 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:10.347 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:10.347 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:10.348 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:10.348 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:10.348 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:10.349 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:10.349 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:10.350 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:10.350 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:10.356 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:10.356 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:10.356 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:10.356 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:10.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:10.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:10.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:29:10.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_not_primary td/ceph-helpers 2026-02-25T22:29:10.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1229: test_get_not_primary: local dir=td/ceph-helpers 2026-02-25T22:29:10.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1231: test_get_not_primary: setup td/ceph-helpers 2026-02-25T22:29:10.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:29:10.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:29:10.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:10.359 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:10.360 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:10.360 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:10.360 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:10.360 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:10.360 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:10.360 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:10.360 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:10.361 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:10.361 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:10.361 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:10.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:10.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:10.363 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:10.364 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:10.364 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:10.364 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:10.364 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:10.365 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:10.367 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:10.367 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:10.368 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:10.368 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:10.368 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:10.368 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:10.369 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:10.369 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:10.369 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:29:10.371 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:29:10.371 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:10.371 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:10.371 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:29:10.372 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:29:10.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:29:10.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:29:10.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:29:10.373 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1232: test_get_not_primary: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-25T22:29:10.374 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:29:10.374 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:29:10.374 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:29:10.374 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:29:10.374 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:29:10.374 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-25T22:29:10.395 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:29:10.395 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:10.395 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:10.395 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:10.395 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:10.395 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:10.396 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:10.396 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-25T22:29:10.423 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:29:10.424 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:29:10.424 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:10.424 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:10.424 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:29:10.424 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:29:10.425 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:10.426 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:10.426 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:10.426 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:10.426 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:10.426 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:10.426 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:10.426 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:10.426 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:29:10.480 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:29:10.480 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:10.480 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:10.480 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:29:10.481 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:29:10.481 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:10.481 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:10.482 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:10.482 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:10.482 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:10.482 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:10.482 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:10.482 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:10.482 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:29:10.536 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1233: test_get_not_primary: run_mgr td/ceph-helpers x 2026-02-25T22:29:10.536 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:29:10.536 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:29:10.536 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:29:10.536 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:29:10.536 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:29:10.536 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:29:10.647 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:29:10.647 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:10.647 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:10.647 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:10.647 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:10.647 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:10.648 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:10.648 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:10.649 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:10.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1234: test_get_not_primary: run_osd td/ceph-helpers 0 2026-02-25T22:29:10.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:29:10.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:10.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:29:10.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:10.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:29:10.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:29:10.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:10.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:10.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:10.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:29:10.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:29:10.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:10.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:10.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:29:10.673 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:10.673 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:10.673 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:10.673 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:10.674 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:10.674 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:10.674 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:10.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:10.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:10.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:10.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:10.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:29:10.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:29:10.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:10.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:10.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:10.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:10.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:10.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:10.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:29:10.675 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:10.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=d03e21d7-0273-44c1-b8c0-141e0cfc9ec8 2026-02-25T22:29:10.676 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 d03e21d7-0273-44c1-b8c0-141e0cfc9ec8' 2026-02-25T22:29:10.676 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 d03e21d7-0273-44c1-b8c0-141e0cfc9ec8 2026-02-25T22:29:10.676 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:10.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC2d59pPcqsKBAAplVFPXHaIZFxXkJIDPf1NQ== 2026-02-25T22:29:10.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC2d59pPcqsKBAAplVFPXHaIZFxXkJIDPf1NQ=="}' 2026-02-25T22:29:10.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new d03e21d7-0273-44c1-b8c0-141e0cfc9ec8 -i td/ceph-helpers/0/new.json 2026-02-25T22:29:10.802 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:10.810 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:29:10.811 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC2d59pPcqsKBAAplVFPXHaIZFxXkJIDPf1NQ== --osd-uuid d03e21d7-0273-44c1-b8c0-141e0cfc9ec8 2026-02-25T22:29:10.827 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:10.818+0000 7fa1a895ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:10.828 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:10.822+0000 7fa1a895ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:10.829 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:10.822+0000 7fa1a895ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:10.830 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:10.822+0000 7fa1a895ca40 -1 bdev(0x55e0c6c34400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:10.830 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:10.822+0000 7fa1a895ca40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:29:11.499 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:29:11.499 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:11.500 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:29:11.500 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:29:11.501 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:11.618 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:29:11.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:29:11.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:11.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:11.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:11.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:11.633 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:11.626+0000 7f4cd0f0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:11.635 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:11.626+0000 7f4cd0f0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:11.636 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:11.630+0000 7f4cd0f0aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:11.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:29:11.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:11.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:29:11.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:11.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:11.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:11.731 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:11.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:11.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:11.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:11.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:12.057 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:12.050+0000 7f4cd0f0aa40 -1 Falling back to public interface 2026-02-25T22:29:12.842 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:12.842 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:12.842 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:29:12.843 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:12.846 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:12.846 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:12.959 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:13.037 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:13.030+0000 7f4cd0f0aa40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:29:13.960 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:13.960 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:13.961 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:29:13.961 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:13.961 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:13.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:14.181 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2472314032,v1:127.0.0.1:6801/2472314032] [v2:127.0.0.1:6802/2472314032,v1:127.0.0.1:6803/2472314032] exists,up d03e21d7-0273-44c1-b8c0-141e0cfc9ec8 2026-02-25T22:29:14.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:14.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:14.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:14.187 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1235: test_get_not_primary: run_osd td/ceph-helpers 1 2026-02-25T22:29:14.187 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:29:14.187 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:14.187 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:29:14.187 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:14.187 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-25T22:29:14.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:29:14.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:14.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:14.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:14.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-25T22:29:14.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-25T22:29:14.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:14.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:14.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:29:14.189 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:14.189 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:14.189 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:14.189 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:14.190 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:14.190 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:14.190 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:14.190 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:14.190 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:14.190 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:14.191 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:14.191 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:29:14.191 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:29:14.191 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:14.191 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:14.191 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:14.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:14.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:14.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:14.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-25T22:29:14.192 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:14.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=185ef3de-e67b-445b-aab6-25d1c5b1b1a4 2026-02-25T22:29:14.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 185ef3de-e67b-445b-aab6-25d1c5b1b1a4' 2026-02-25T22:29:14.193 INFO:tasks.workunit.client.0.trial188.stdout:add osd1 185ef3de-e67b-445b-aab6-25d1c5b1b1a4 2026-02-25T22:29:14.193 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:14.199 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQC6d59pJtyqCxAA4AXZVcM3tIN/cHa03enYFw== 2026-02-25T22:29:14.199 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQC6d59pJtyqCxAA4AXZVcM3tIN/cHa03enYFw=="}' 2026-02-25T22:29:14.199 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 185ef3de-e67b-445b-aab6-25d1c5b1b1a4 -i td/ceph-helpers/1/new.json 2026-02-25T22:29:14.417 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:29:14.426 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-25T22:29:14.427 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQC6d59pJtyqCxAA4AXZVcM3tIN/cHa03enYFw== --osd-uuid 185ef3de-e67b-445b-aab6-25d1c5b1b1a4 2026-02-25T22:29:14.443 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:14.433+0000 7f93e8253a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:14.445 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:14.437+0000 7f93e8253a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:14.446 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:14.437+0000 7f93e8253a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:14.447 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:14.437+0000 7f93e8253a40 -1 bdev(0x564710c5e400 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:14.447 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:14.437+0000 7f93e8253a40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-25T22:29:15.091 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-25T22:29:15.091 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:15.092 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:29:15.092 INFO:tasks.workunit.client.0.trial188.stdout:adding osd1 key to auth repository 2026-02-25T22:29:15.093 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:15.402 INFO:tasks.workunit.client.0.trial188.stdout:start osd.1 2026-02-25T22:29:15.403 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:29:15.403 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:15.403 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:15.403 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:15.403 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:15.417 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:15.409+0000 7feb24e06a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:15.418 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:15.409+0000 7feb24e06a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:15.419 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:15.409+0000 7feb24e06a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:15.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:29:15.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:15.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:29:15.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:15.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:15.621 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:15.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:15.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:15.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:15.622 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:29:15.837 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:15.849 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:15.841+0000 7feb24e06a40 -1 Falling back to public interface 2026-02-25T22:29:16.412 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:16.405+0000 7feb24e06a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:29:16.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:16.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:16.839 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:29:16.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:16.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:16.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:29:17.050 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:18.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:18.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:18.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:18.053 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:29:18.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:18.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:29:18.267 INFO:tasks.workunit.client.0.trial188.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/487701105,v1:127.0.0.1:6811/487701105] [v2:127.0.0.1:6812/487701105,v1:127.0.0.1:6813/487701105] exists,up 185ef3de-e67b-445b-aab6-25d1c5b1b1a4 2026-02-25T22:29:18.268 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:18.268 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:18.268 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:18.268 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1236: test_get_not_primary: create_rbd_pool 2026-02-25T22:29:18.269 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:29:18.467 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:29:18.476 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:29:18.476 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:29:18.710 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:29:18.720 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:29:19.721 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:29:20.033 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1237: test_get_not_primary: wait_for_clean 2026-02-25T22:29:20.033 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:29:20.033 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:29:20.033 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:29:20.034 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:29:20.034 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:29:20.034 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:29:20.034 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:29:20.034 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:29:20.034 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:29:20.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:29:20.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:29:20.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:29:20.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:29:20.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:29:20.131 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:29:20.353 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:29:20.353 INFO:tasks.workunit.client.0.trial188.stderr:1' 2026-02-25T22:29:20.353 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:29:20.353 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:20.353 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:29:20.432 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-25T22:29:20.432 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-25T22:29:20.433 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836484' 2026-02-25T22:29:20.433 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:20.433 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:29:20.514 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-25T22:29:20.514 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-25T22:29:20.514 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836484 1-38654705667' 2026-02-25T22:29:20.514 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:20.515 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836484 2026-02-25T22:29:20.515 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:20.515 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:29:20.516 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-25T22:29:20.516 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:20.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836484 2026-02-25T22:29:20.518 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-25T22:29:20.518 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836484 2026-02-25T22:29:20.518 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:20.737 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836484 2026-02-25T22:29:20.737 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:29:21.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:29:21.738 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:21.959 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-25T22:29:21.959 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:21.959 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705667 2026-02-25T22:29:21.959 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:21.960 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:29:21.961 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-25T22:29:21.961 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:21.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705667 2026-02-25T22:29:21.962 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.1 seq 38654705667 2026-02-25T22:29:21.963 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-25T22:29:21.963 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:29:22.183 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-25T22:29:22.184 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:29:22.184 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:22.184 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:22.474 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:29:22.474 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:29:22.474 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:29:22.474 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:29:22.474 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:29:22.474 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:29:22.475 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:29:22.475 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:29:22.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:29:22.683 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:29:22.683 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:22.683 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:22.961 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:29:22.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:29:22.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:29:22.962 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: get_primary rbd GROUP 2026-02-25T22:29:22.962 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-25T22:29:22.962 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2026-02-25T22:29:22.962 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2026-02-25T22:29:22.962 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-25T22:29:23.179 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1238: test_get_not_primary: local primary=1 2026-02-25T22:29:23.179 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: test_get_not_primary: get_not_primary rbd GROUP 2026-02-25T22:29:23.180 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1220: get_not_primary: local poolname=rbd 2026-02-25T22:29:23.180 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1221: get_not_primary: local objectname=GROUP 2026-02-25T22:29:23.180 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: get_primary rbd GROUP 2026-02-25T22:29:23.180 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-25T22:29:23.180 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2026-02-25T22:29:23.180 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2026-02-25T22:29:23.180 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-25T22:29:23.397 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1223: get_not_primary: local primary=1 2026-02-25T22:29:23.398 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1224: get_not_primary: ceph --format json osd map rbd GROUP 2026-02-25T22:29:23.398 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1225: get_not_primary: jq '.acting | map(select (. != 1)) | .[0]' 2026-02-25T22:29:23.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1239: test_get_not_primary: local not_primary=0 2026-02-25T22:29:23.615 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1240: test_get_not_primary: test 0 '!=' 1 2026-02-25T22:29:23.615 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1241: test_get_not_primary: test 0 = 0 -o 0 = 1 2026-02-25T22:29:23.615 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1242: test_get_not_primary: teardown td/ceph-helpers 2026-02-25T22:29:23.615 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:23.615 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:23.615 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:23.615 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:23.615 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:23.615 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:23.615 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:23.615 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:23.723 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:23.723 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:23.724 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:23.724 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:23.726 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:23.726 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:23.726 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:23.727 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:23.727 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:23.728 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:23.728 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:23.729 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:23.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:23.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:23.737 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:23.737 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:23.737 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:23.737 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:23.739 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:23.739 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:23.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:29:23.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_num_active_clean td/ceph-helpers 2026-02-25T22:29:23.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1363: test_get_num_active_clean: local dir=td/ceph-helpers 2026-02-25T22:29:23.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1365: test_get_num_active_clean: setup td/ceph-helpers 2026-02-25T22:29:23.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:29:23.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:29:23.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:23.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:23.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:23.740 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:23.740 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:23.740 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:23.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:23.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:23.741 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:23.742 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:23.743 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:23.743 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:23.744 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:23.744 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:23.745 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:23.746 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:23.746 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:23.746 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:23.746 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:23.747 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:23.749 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:23.749 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:23.750 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:23.750 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:23.750 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:23.750 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:23.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:23.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:23.752 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:29:23.753 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:29:23.753 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:23.753 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:23.753 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:29:23.755 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:29:23.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:29:23.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:29:23.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:29:23.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1366: test_get_num_active_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:23.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:29:23.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:29:23.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:29:23.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:29:23.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:29:23.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:23.777 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:29:23.777 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:23.777 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:23.778 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:23.778 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:23.778 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:23.778 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:23.779 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:23.806 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:29:23.807 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:29:23.807 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:23.807 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:23.807 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:29:23.809 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:29:23.809 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:23.809 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:23.809 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:23.809 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:23.809 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:23.809 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:23.809 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:23.809 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:23.810 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:29:23.863 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:29:23.863 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:23.863 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:23.864 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:29:23.865 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:29:23.865 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:23.865 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:23.865 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:23.865 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:23.865 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:23.865 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:23.865 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:23.865 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:23.866 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:29:23.918 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1367: test_get_num_active_clean: run_mgr td/ceph-helpers x 2026-02-25T22:29:23.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:29:23.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:29:23.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:29:23.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:29:23.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:29:23.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:29:24.029 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:29:24.029 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:24.029 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:24.030 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:24.030 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:24.030 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:24.031 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:24.031 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:24.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:24.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1368: test_get_num_active_clean: run_osd td/ceph-helpers 0 2026-02-25T22:29:24.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:29:24.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:24.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:29:24.052 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:24.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:29:24.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:29:24.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:24.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:24.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:24.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:29:24.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:29:24.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:24.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:24.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:29:24.053 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:24.053 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:24.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:24.055 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:24.055 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:24.055 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:24.055 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:29:24.055 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:24.056 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=dc6e376b-5a00-4bfd-b30f-8aa158b70ef3 2026-02-25T22:29:24.056 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 dc6e376b-5a00-4bfd-b30f-8aa158b70ef3' 2026-02-25T22:29:24.056 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 dc6e376b-5a00-4bfd-b30f-8aa158b70ef3 2026-02-25T22:29:24.056 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:24.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDEd59pdTvFAxAAx+I3p1wMQOso1iIQ3RQrHQ== 2026-02-25T22:29:24.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDEd59pdTvFAxAAx+I3p1wMQOso1iIQ3RQrHQ=="}' 2026-02-25T22:29:24.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new dc6e376b-5a00-4bfd-b30f-8aa158b70ef3 -i td/ceph-helpers/0/new.json 2026-02-25T22:29:24.180 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:24.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:29:24.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDEd59pdTvFAxAAx+I3p1wMQOso1iIQ3RQrHQ== --osd-uuid dc6e376b-5a00-4bfd-b30f-8aa158b70ef3 2026-02-25T22:29:24.206 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:24.201+0000 7f6957e46a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:24.207 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:24.201+0000 7f6957e46a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:24.208 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:24.205+0000 7f6957e46a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:24.209 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:24.205+0000 7f6957e46a40 -1 bdev(0x55b9bf900400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:24.209 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:24.205+0000 7f6957e46a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:29:24.932 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:29:24.932 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:24.933 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:29:24.934 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:29:24.934 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:25.054 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:29:25.055 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:29:25.056 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:25.056 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:25.056 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:25.056 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:25.079 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:25.073+0000 7f8376e72a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:25.082 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:25.077+0000 7f8376e72a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:25.083 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:25.077+0000 7f8376e72a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:25.166 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:29:25.166 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:25.166 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:29:25.166 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:25.166 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:25.166 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:25.167 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:25.167 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:25.168 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:25.168 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:25.273 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:25.625 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:25.621+0000 7f8376e72a40 -1 Falling back to public interface 2026-02-25T22:29:26.275 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:26.275 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:29:26.275 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:26.276 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:26.276 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:26.276 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:26.385 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:26.616 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:26.609+0000 7f8376e72a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:29:27.386 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:27.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:27.387 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:29:27.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:27.388 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:27.388 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:27.599 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3442257514,v1:127.0.0.1:6801/3442257514] [v2:127.0.0.1:6802/3442257514,v1:127.0.0.1:6803/3442257514] exists,up dc6e376b-5a00-4bfd-b30f-8aa158b70ef3 2026-02-25T22:29:27.600 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:27.600 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:27.600 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:27.600 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1369: test_get_num_active_clean: create_rbd_pool 2026-02-25T22:29:27.600 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:29:27.799 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:29:27.809 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:29:27.809 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:29:28.043 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:29:28.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:29:29.055 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:29:29.331 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1370: test_get_num_active_clean: wait_for_clean 2026-02-25T22:29:29.332 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:29:29.332 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:29:29.332 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:29:29.332 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:29:29.333 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:29:29.333 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:29:29.333 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:29:29.333 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:29:29.333 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:29:29.431 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:29:29.431 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:29:29.431 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:29:29.431 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:29:29.431 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:29:29.431 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:29:29.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:29:29.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:29:29.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:29.652 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:29:29.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:29:29.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:29:29.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:29:29.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:29.732 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:29:29.732 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:29.732 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:29:29.733 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:29:29.733 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:29.734 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:29:29.734 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:29:29.735 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:29:29.735 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:29.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:29:29.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:29:30.963 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:29:30.964 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:31.193 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:29:31.194 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:29:31.194 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:31.194 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:31.481 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:29:31.481 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:29:31.481 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:29:31.481 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:29:31.482 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:29:31.482 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:29:31.482 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:29:31.482 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:29:31.696 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:29:31.697 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:29:31.697 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:31.697 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:31.989 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:29:31.989 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:29:31.989 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:29:31.989 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1371: test_get_num_active_clean: get_num_active_clean 2026-02-25T22:29:31.989 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:29:31.990 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:29:31.990 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:29:31.990 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:29:31.990 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:29:32.205 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1371: test_get_num_active_clean: local num_active_clean=4 2026-02-25T22:29:32.205 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1372: test_get_num_active_clean: test 4 = 4 2026-02-25T22:29:32.205 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1373: test_get_num_active_clean: teardown td/ceph-helpers 2026-02-25T22:29:32.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:32.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:32.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:32.206 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:32.206 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:32.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:32.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:32.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:32.313 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:32.314 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:32.314 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:32.314 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:32.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:32.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:32.316 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:32.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:32.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:32.318 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:32.318 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:32.319 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:32.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:32.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:32.324 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:32.324 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:32.324 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:32.324 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:32.326 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:32.326 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:32.326 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:29:32.326 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_num_active_or_peered td/ceph-helpers 2026-02-25T22:29:32.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1393: test_get_num_active_or_peered: local dir=td/ceph-helpers 2026-02-25T22:29:32.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1395: test_get_num_active_or_peered: setup td/ceph-helpers 2026-02-25T22:29:32.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:29:32.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:29:32.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:32.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:32.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:32.327 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:32.327 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:32.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:32.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:32.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:32.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:32.329 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:32.329 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:32.329 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:32.330 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:32.330 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:32.331 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:32.331 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:32.331 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:32.332 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:32.332 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:32.333 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:32.334 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:32.334 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:32.335 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:32.335 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:32.335 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:32.335 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:32.336 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:32.337 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:32.337 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:29:32.338 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:29:32.338 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:32.338 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:32.338 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:29:32.340 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:29:32.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:29:32.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:29:32.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:29:32.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1396: test_get_num_active_or_peered: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:32.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:29:32.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:29:32.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:29:32.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:29:32.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:29:32.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:32.363 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:29:32.363 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:32.363 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:32.364 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:32.364 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:32.364 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:32.364 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:32.364 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:32.393 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:29:32.394 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:29:32.394 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:32.394 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:32.394 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:29:32.395 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:29:32.395 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:32.395 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:32.395 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:32.395 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:32.396 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:32.396 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:32.396 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:32.396 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:32.396 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:29:32.452 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:29:32.452 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:32.452 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:32.452 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:29:32.453 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:29:32.454 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:32.454 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:32.454 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:32.454 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:32.454 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:32.454 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:32.454 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:32.454 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:32.454 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:29:32.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1397: test_get_num_active_or_peered: run_mgr td/ceph-helpers x 2026-02-25T22:29:32.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:29:32.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:29:32.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:29:32.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:29:32.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:29:32.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:29:32.626 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:29:32.626 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:32.627 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:32.627 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:32.627 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:32.627 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:32.627 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:32.628 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:32.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:32.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1398: test_get_num_active_or_peered: run_osd td/ceph-helpers 0 2026-02-25T22:29:32.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:29:32.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:32.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:29:32.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:32.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:29:32.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:29:32.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:32.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:32.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:32.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:29:32.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:29:32.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:32.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:32.651 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:29:32.652 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:32.652 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:32.652 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:32.652 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:32.652 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:32.652 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:32.652 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:32.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:32.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:32.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:32.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:32.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:29:32.653 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:29:32.653 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:32.653 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:32.653 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:32.653 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:32.653 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:32.653 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:32.653 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:29:32.653 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:32.654 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=40264104-2b47-4e6b-aa40-869cadd0f2e7 2026-02-25T22:29:32.654 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 40264104-2b47-4e6b-aa40-869cadd0f2e7' 2026-02-25T22:29:32.654 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 40264104-2b47-4e6b-aa40-869cadd0f2e7 2026-02-25T22:29:32.654 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:32.663 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDMd59pd/9aJxAAUtiGdQF2w9+NEm8qOwgLIg== 2026-02-25T22:29:32.663 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDMd59pd/9aJxAAUtiGdQF2w9+NEm8qOwgLIg=="}' 2026-02-25T22:29:32.663 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 40264104-2b47-4e6b-aa40-869cadd0f2e7 -i td/ceph-helpers/0/new.json 2026-02-25T22:29:32.772 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:32.779 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:29:32.780 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDMd59pd/9aJxAAUtiGdQF2w9+NEm8qOwgLIg== --osd-uuid 40264104-2b47-4e6b-aa40-869cadd0f2e7 2026-02-25T22:29:32.797 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:32.793+0000 7fa02d442a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:32.798 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:32.793+0000 7fa02d442a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:32.799 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:32.793+0000 7fa02d442a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:32.800 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:32.797+0000 7fa02d442a40 -1 bdev(0x55f134f86400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:32.800 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:32.797+0000 7fa02d442a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:29:33.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:29:33.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:33.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:29:33.504 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:29:33.504 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:33.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:29:33.620 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:29:33.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:33.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:33.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:33.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:33.636 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:33.633+0000 7f10692f7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:33.637 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:33.633+0000 7f10692f7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:33.638 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:33.633+0000 7f10692f7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:33.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:29:33.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:33.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:29:33.731 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:33.732 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:33.732 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:33.732 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:33.732 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:33.733 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:33.733 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:33.844 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:34.165 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:34.161+0000 7f10692f7a40 -1 Falling back to public interface 2026-02-25T22:29:34.846 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:34.847 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:34.847 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:34.847 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:29:34.848 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:34.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:34.973 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:35.165 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:35.161+0000 7f10692f7a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:29:35.974 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:35.974 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:35.975 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:29:35.976 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:35.976 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:35.976 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:36.190 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/588929702,v1:127.0.0.1:6801/588929702] [v2:127.0.0.1:6802/588929702,v1:127.0.0.1:6803/588929702] exists,up 40264104-2b47-4e6b-aa40-869cadd0f2e7 2026-02-25T22:29:36.191 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:36.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:36.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:36.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1399: test_get_num_active_or_peered: create_rbd_pool 2026-02-25T22:29:36.192 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:29:36.402 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:29:36.412 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:29:36.412 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:29:36.692 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:29:36.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:29:37.704 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:29:38.012 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1400: test_get_num_active_or_peered: wait_for_clean 2026-02-25T22:29:38.012 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:29:38.012 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:29:38.012 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:29:38.012 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:29:38.013 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:29:38.013 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:29:38.013 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:29:38.014 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:29:38.014 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:29:38.107 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:29:38.107 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:29:38.107 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:29:38.107 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:29:38.107 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:29:38.107 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:29:38.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:29:38.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:29:38.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:38.317 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:29:38.399 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:29:38.399 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:29:38.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:29:38.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:38.400 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:29:38.400 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:38.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:29:38.401 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:29:38.402 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:38.403 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:29:38.403 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:29:38.403 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:29:38.404 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:38.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:29:38.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:29:39.622 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:29:39.622 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:39.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:29:39.841 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:29:39.841 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:39.842 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:40.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:29:40.126 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:29:40.126 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:29:40.126 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:29:40.126 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:29:40.126 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:29:40.127 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:29:40.127 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:29:40.334 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:29:40.334 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:29:40.334 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:40.335 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:40.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:29:40.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:29:40.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:29:40.620 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1401: test_get_num_active_or_peered: get_num_active_or_peered 2026-02-25T22:29:40.620 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1385: get_num_active_or_peered: local expression 2026-02-25T22:29:40.621 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1386: get_num_active_or_peered: expression+='select(contains("active") or contains("peered")) | ' 2026-02-25T22:29:40.621 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1387: get_num_active_or_peered: expression+='select(contains("stale") | not)' 2026-02-25T22:29:40.621 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1388: get_num_active_or_peered: ceph --format json pg dump pgs 2026-02-25T22:29:40.621 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1389: get_num_active_or_peered: jq '.pg_stats | [.[] | .state | select(contains("active") or contains("peered")) | select(contains("stale") | not)] | length' 2026-02-25T22:29:40.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1401: test_get_num_active_or_peered: local num_peered=4 2026-02-25T22:29:40.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1402: test_get_num_active_or_peered: test 4 = 4 2026-02-25T22:29:40.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1403: test_get_num_active_or_peered: teardown td/ceph-helpers 2026-02-25T22:29:40.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:40.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:40.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:40.827 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:40.827 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:40.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:40.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:40.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:40.934 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:40.935 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:40.936 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:40.936 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:40.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:40.938 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:40.938 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:40.939 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:40.939 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:40.939 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:40.940 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:40.940 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:40.941 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:40.941 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:40.946 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:40.947 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:40.947 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:40.947 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:40.948 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_num_pgs td/ceph-helpers 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1420: test_get_num_pgs: local dir=td/ceph-helpers 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1422: test_get_num_pgs: setup td/ceph-helpers 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:40.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:40.951 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:40.952 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:40.953 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:40.953 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:40.954 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:40.954 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:40.954 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:40.956 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:40.956 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:40.956 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:40.956 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:40.957 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:40.958 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:40.958 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:40.959 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:40.959 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:40.959 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:40.960 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:40.960 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:40.960 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:40.960 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:29:40.962 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:29:40.962 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:40.962 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:40.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:29:40.964 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:29:40.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:29:40.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:29:40.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:29:40.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1423: test_get_num_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:40.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:29:40.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:29:40.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:29:40.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:29:40.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:29:40.965 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:40.987 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:29:40.987 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:40.987 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:40.988 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:40.988 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:40.988 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:40.988 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:40.988 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:41.019 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:29:41.020 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:29:41.020 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:41.020 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:41.020 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:29:41.021 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:29:41.021 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:41.021 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:41.021 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:41.022 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:41.022 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:41.022 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:41.022 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:41.022 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:41.022 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:29:41.077 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:29:41.077 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:41.077 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:41.077 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:29:41.077 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:29:41.077 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:41.077 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:41.077 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:41.078 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:41.078 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:41.078 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:41.078 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:41.078 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:41.078 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:29:41.132 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1424: test_get_num_pgs: run_mgr td/ceph-helpers x 2026-02-25T22:29:41.132 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:29:41.132 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:29:41.132 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:29:41.132 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:29:41.132 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:29:41.132 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:29:41.243 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:29:41.243 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:41.243 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:41.244 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:41.244 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:41.244 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:41.244 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:41.245 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:41.246 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1425: test_get_num_pgs: run_osd td/ceph-helpers 0 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:41.265 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:41.266 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:41.266 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:41.266 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:41.266 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:41.266 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:41.266 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:41.266 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:41.266 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:41.266 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:29:41.267 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:29:41.267 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:41.267 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:41.267 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:41.267 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:41.267 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:41.267 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:41.267 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:29:41.267 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:41.268 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0f4c1e2d-46b9-46d2-a2b6-5d8d5e645f9b 2026-02-25T22:29:41.268 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0f4c1e2d-46b9-46d2-a2b6-5d8d5e645f9b' 2026-02-25T22:29:41.268 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 0f4c1e2d-46b9-46d2-a2b6-5d8d5e645f9b 2026-02-25T22:29:41.268 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:41.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDVd59pMruQEBAA8amcYX4hLmbFkW4rx8rABQ== 2026-02-25T22:29:41.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDVd59pMruQEBAA8amcYX4hLmbFkW4rx8rABQ=="}' 2026-02-25T22:29:41.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0f4c1e2d-46b9-46d2-a2b6-5d8d5e645f9b -i td/ceph-helpers/0/new.json 2026-02-25T22:29:41.392 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:41.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:29:41.401 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDVd59pMruQEBAA8amcYX4hLmbFkW4rx8rABQ== --osd-uuid 0f4c1e2d-46b9-46d2-a2b6-5d8d5e645f9b 2026-02-25T22:29:41.417 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:41.413+0000 7fdccd254a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:41.419 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:41.413+0000 7fdccd254a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:41.419 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:41.413+0000 7fdccd254a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:41.420 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:41.417+0000 7fdccd254a40 -1 bdev(0x565317344400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:41.420 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:41.417+0000 7fdccd254a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:29:42.218 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:29:42.219 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:42.219 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:29:42.220 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:29:42.220 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:42.341 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:29:42.341 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:29:42.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:42.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:42.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:42.342 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:42.365 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:42.361+0000 7f79494a4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:42.368 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:42.365+0000 7f79494a4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:42.369 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:42.365+0000 7f79494a4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:42.448 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:29:42.449 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:42.449 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:29:42.449 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:42.449 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:42.449 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:42.449 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:42.450 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:42.450 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:42.450 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:42.556 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:42.825 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:42.821+0000 7f79494a4a40 -1 Falling back to public interface 2026-02-25T22:29:43.557 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:43.558 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:43.558 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:29:43.558 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:43.559 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:43.559 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:43.664 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:43.833 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:43.829+0000 7f79494a4a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:29:44.666 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:44.666 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:44.666 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:29:44.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:44.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:44.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:44.881 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1218932052,v1:127.0.0.1:6801/1218932052] [v2:127.0.0.1:6802/1218932052,v1:127.0.0.1:6803/1218932052] exists,up 0f4c1e2d-46b9-46d2-a2b6-5d8d5e645f9b 2026-02-25T22:29:44.882 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:44.882 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:44.882 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:44.883 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1426: test_get_num_pgs: create_rbd_pool 2026-02-25T22:29:44.883 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:29:45.093 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:29:45.105 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:29:45.105 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:29:45.325 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:29:45.335 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:29:46.336 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:29:46.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1427: test_get_num_pgs: wait_for_clean 2026-02-25T22:29:46.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:29:46.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:29:46.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:29:46.630 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:29:46.630 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:29:46.630 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:29:46.630 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:29:46.631 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:29:46.631 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:29:46.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:29:46.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:29:46.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:29:46.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:29:46.730 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:29:46.730 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:29:46.947 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:29:46.948 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:29:46.948 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:29:46.948 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:29:47.027 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:29:47.027 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:29:47.027 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:29:47.027 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:29:47.028 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:29:47.028 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:29:47.029 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:29:47.030 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:29:47.030 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:29:47.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:29:47.031 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:29:47.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:29:47.032 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:47.257 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:29:47.257 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:29:48.258 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:29:48.259 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:29:48.479 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:29:48.479 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:29:48.480 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:48.480 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:48.773 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:29:48.773 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:29:48.774 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:29:48.774 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:29:48.774 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:29:48.774 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:29:48.774 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:29:48.774 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:29:48.990 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:29:48.990 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:29:48.990 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:48.991 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:49.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:29:49.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:29:49.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:29:49.270 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1428: test_get_num_pgs: get_num_pgs 2026-02-25T22:29:49.270 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:29:49.271 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:29:49.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1428: test_get_num_pgs: local num_pgs=4 2026-02-25T22:29:49.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1429: test_get_num_pgs: test 4 -gt 0 2026-02-25T22:29:49.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1430: test_get_num_pgs: teardown td/ceph-helpers 2026-02-25T22:29:49.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:49.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:49.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:49.550 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:49.550 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:49.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:49.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:49.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:49.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:49.658 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:49.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:49.659 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:49.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:49.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:49.660 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:49.662 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:49.662 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:49.662 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:49.662 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:49.663 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:49.664 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:49.664 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:49.670 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:49.670 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:49.670 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:49.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:49.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:49.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:49.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:29:49.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_op_scheduler td/ceph-helpers 2026-02-25T22:29:49.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2315: test_get_op_scheduler: local dir=td/ceph-helpers 2026-02-25T22:29:49.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2317: test_get_op_scheduler: setup td/ceph-helpers 2026-02-25T22:29:49.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:29:49.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:29:49.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:49.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:49.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:49.674 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:49.674 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:49.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:49.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:49.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:49.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:49.675 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:49.676 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:49.676 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:49.677 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:49.677 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:49.677 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:49.678 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:49.679 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:49.679 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:49.679 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:49.680 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:49.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:49.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:49.682 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:49.682 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:49.683 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:49.683 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:49.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:49.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:49.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:29:49.685 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:29:49.685 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:49.685 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:49.685 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:29:49.687 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:29:49.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:29:49.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:29:49.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:29:49.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2319: test_get_op_scheduler: run_mon td/ceph-helpers a 2026-02-25T22:29:49.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:29:49.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:29:49.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:29:49.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:29:49.688 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:29:49.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers 2026-02-25T22:29:49.709 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:29:49.709 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:49.709 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:49.710 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:49.710 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:49.710 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:49.710 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:49.710 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false 2026-02-25T22:29:49.738 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:29:49.739 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:29:49.739 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:49.739 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:49.740 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:29:49.741 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:29:49.741 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:49.741 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:49.741 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:49.741 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:49.741 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:49.741 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:49.741 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:49.742 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:49.742 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:29:49.797 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:29:49.797 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:49.798 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:49.798 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:29:49.799 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:29:49.799 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:49.799 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:49.799 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:49.799 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:49.799 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:49.799 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:49.799 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:49.800 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:49.800 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:29:49.855 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2320: test_get_op_scheduler: run_mgr td/ceph-helpers x 2026-02-25T22:29:49.855 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:29:49.855 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:29:49.855 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:29:49.855 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:29:49.855 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:29:49.855 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:29:49.969 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:29:49.970 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:49.970 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:49.970 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:49.971 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:49.971 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:49.971 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:49.971 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:49.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:49.992 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2322: test_get_op_scheduler: run_osd td/ceph-helpers 0 --osd_op_queue=wpq 2026-02-25T22:29:49.992 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:29:49.992 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:49.992 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:29:49.992 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:49.992 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:29:49.992 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:29:49.993 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:49.993 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:49.993 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:49.993 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:29:49.993 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:29:49.993 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:49.993 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:49.993 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:29:49.993 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:49.993 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:49.993 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:49.993 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:49.994 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:49.994 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:49.994 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:49.994 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:49.994 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:49.994 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:49.994 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:49.994 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:29:49.994 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:29:49.994 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:49.994 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:49.994 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:49.994 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:49.995 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:49.995 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=wpq 2026-02-25T22:29:49.995 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:29:49.995 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:49.995 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=8f661628-3d1e-4142-80ee-b65776374ebc 2026-02-25T22:29:49.996 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 8f661628-3d1e-4142-80ee-b65776374ebc' 2026-02-25T22:29:49.996 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 8f661628-3d1e-4142-80ee-b65776374ebc 2026-02-25T22:29:49.996 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:50.005 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDed59pxBAXABAAteF1zclUFyIftu8cEWbEeA== 2026-02-25T22:29:50.005 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDed59pxBAXABAAteF1zclUFyIftu8cEWbEeA=="}' 2026-02-25T22:29:50.005 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 8f661628-3d1e-4142-80ee-b65776374ebc -i td/ceph-helpers/0/new.json 2026-02-25T22:29:50.113 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:50.121 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:29:50.122 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq --mkfs --key AQDed59pxBAXABAAteF1zclUFyIftu8cEWbEeA== --osd-uuid 8f661628-3d1e-4142-80ee-b65776374ebc 2026-02-25T22:29:50.138 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:50.133+0000 7f6319853a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:50.140 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:50.133+0000 7f6319853a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:50.140 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:50.137+0000 7f6319853a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:50.141 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:50.137+0000 7f6319853a40 -1 bdev(0x564d14c94400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:50.141 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:50.137+0000 7f6319853a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:29:50.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:29:50.827 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:50.828 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:29:50.829 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:29:50.829 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:50.951 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:29:50.952 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:29:50.952 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=wpq 2026-02-25T22:29:50.952 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:50.952 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:50.952 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:50.975 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:50.969+0000 7f13fd6f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:50.977 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:50.973+0000 7f13fd6f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:50.979 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:50.973+0000 7f13fd6f9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:51.064 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:29:51.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:51.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:29:51.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:51.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:51.065 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:51.065 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:51.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:51.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:51.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:51.174 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:51.562 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:51.557+0000 7f13fd6f9a40 -1 Falling back to public interface 2026-02-25T22:29:52.175 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:52.176 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:52.176 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:52.176 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:29:52.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:52.181 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:52.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:52.333 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:52.329+0000 7f13fd6f9a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:29:53.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:53.318 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:53.318 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:29:53.319 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:29:53.319 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:53.319 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:53.533 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3875227615,v1:127.0.0.1:6801/3875227615] [v2:127.0.0.1:6802/3875227615,v1:127.0.0.1:6803/3875227615] exists,up 8f661628-3d1e-4142-80ee-b65776374ebc 2026-02-25T22:29:53.534 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:53.534 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:53.534 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:53.535 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: test_get_op_scheduler: get_op_scheduler 0 2026-02-25T22:29:53.535 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: get_op_scheduler: local id=0 2026-02-25T22:29:53.535 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: get_op_scheduler: get_config osd 0 osd_op_queue 2026-02-25T22:29:53.535 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2026-02-25T22:29:53.535 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=0 2026-02-25T22:29:53.535 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_op_queue 2026-02-25T22:29:53.535 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_op_queue 2026-02-25T22:29:53.536 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.0 2026-02-25T22:29:53.536 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.0 2026-02-25T22:29:53.536 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.0 ']' 2026-02-25T22:29:53.537 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:53.537 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:53.537 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:53.537 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-osd.0.asok 2026-02-25T22:29:53.537 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:53.537 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-osd.0.asok config get osd_op_queue 2026-02-25T22:29:53.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2323: test_get_op_scheduler: test wpq = wpq 2026-02-25T22:29:53.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2325: test_get_op_scheduler: run_osd td/ceph-helpers 1 --osd_op_queue=mclock_scheduler 2026-02-25T22:29:53.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:29:53.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:53.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:29:53.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:53.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-25T22:29:53.601 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:29:53.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:53.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:53.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:53.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-25T22:29:53.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-25T22:29:53.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:53.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:53.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:29:53.602 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:53.602 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:53.602 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:53.602 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:53.602 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:53.603 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:53.603 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:53.603 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:53.603 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:53.603 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:53.603 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:53.603 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:29:53.603 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:29:53.603 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:53.603 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:53.603 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:53.603 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:53.603 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:53.604 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+=--osd_op_queue=mclock_scheduler 2026-02-25T22:29:53.604 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-25T22:29:53.604 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:53.605 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a32221df-bb4e-4828-b407-925c9eab35c2 2026-02-25T22:29:53.605 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 a32221df-bb4e-4828-b407-925c9eab35c2' 2026-02-25T22:29:53.605 INFO:tasks.workunit.client.0.trial188.stdout:add osd1 a32221df-bb4e-4828-b407-925c9eab35c2 2026-02-25T22:29:53.605 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:53.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDhd59pVG9rJBAALlrD2igkMxSpTLZgPSzc7A== 2026-02-25T22:29:53.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDhd59pVG9rJBAALlrD2igkMxSpTLZgPSzc7A=="}' 2026-02-25T22:29:53.614 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a32221df-bb4e-4828-b407-925c9eab35c2 -i td/ceph-helpers/1/new.json 2026-02-25T22:29:53.836 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:29:53.845 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-25T22:29:53.846 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler --mkfs --key AQDhd59pVG9rJBAALlrD2igkMxSpTLZgPSzc7A== --osd-uuid a32221df-bb4e-4828-b407-925c9eab35c2 2026-02-25T22:29:53.862 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:53.857+0000 7f774a3d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:53.863 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:53.857+0000 7f774a3d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:53.864 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:53.861+0000 7f774a3d8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:53.865 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:53.861+0000 7f774a3d8a40 -1 bdev(0x5649e485a400 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:53.865 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:53.861+0000 7f774a3d8a40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-25T22:29:54.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-25T22:29:54.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:54.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:29:54.503 INFO:tasks.workunit.client.0.trial188.stdout:adding osd1 key to auth repository 2026-02-25T22:29:54.504 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:54.793 INFO:tasks.workunit.client.0.trial188.stdout:start osd.1 2026-02-25T22:29:54.794 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:29:54.794 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --osd_op_queue=mclock_scheduler 2026-02-25T22:29:54.794 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:54.794 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:54.795 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:54.817 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:54.813+0000 7f6f6c8f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:54.819 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:54.813+0000 7f6f6c8f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:54.820 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:54.813+0000 7f6f6c8f8a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:55.012 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:29:55.013 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:55.013 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:29:55.013 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:55.013 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:55.013 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:55.013 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:55.014 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:55.014 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:55.014 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:29:55.153 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:55.149+0000 7f6f6c8f8a40 -1 Falling back to public interface 2026-02-25T22:29:55.228 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:55.781 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:55.777+0000 7f6f6c8f8a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:29:56.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:56.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:56.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:56.230 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:29:56.230 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:56.230 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:29:56.445 INFO:tasks.workunit.client.0.trial188.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2559692323,v1:127.0.0.1:6811/2559692323] [v2:127.0.0.1:6812/2559692323,v1:127.0.0.1:6813/2559692323] exists,up a32221df-bb4e-4828-b407-925c9eab35c2 2026-02-25T22:29:56.446 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:29:56.447 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:29:56.447 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:29:56.447 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: test_get_op_scheduler: get_op_scheduler 1 2026-02-25T22:29:56.447 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2309: get_op_scheduler: local id=1 2026-02-25T22:29:56.447 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2311: get_op_scheduler: get_config osd 1 osd_op_queue 2026-02-25T22:29:56.447 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=osd 2026-02-25T22:29:56.447 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=1 2026-02-25T22:29:56.447 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=osd_op_queue 2026-02-25T22:29:56.447 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .osd_op_queue 2026-02-25T22:29:56.448 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path osd.1 2026-02-25T22:29:56.449 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=osd.1 2026-02-25T22:29:56.449 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n osd.1 ']' 2026-02-25T22:29:56.449 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:56.449 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:56.449 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:56.449 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-osd.1.asok 2026-02-25T22:29:56.449 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:56.449 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-osd.1.asok config get osd_op_queue 2026-02-25T22:29:56.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2326: test_get_op_scheduler: test mclock_scheduler = mclock_scheduler 2026-02-25T22:29:56.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2327: test_get_op_scheduler: teardown td/ceph-helpers 2026-02-25T22:29:56.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:56.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:56.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:56.510 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:56.510 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:56.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:56.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:56.510 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:56.618 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:56.618 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:56.619 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:56.619 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:56.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:56.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:56.621 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:56.622 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:56.622 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:56.623 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:56.623 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:56.623 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:56.625 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:56.625 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:56.631 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:56.631 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:56.631 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:56.631 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:56.633 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:56.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:56.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:29:56.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_osd_id_used_by_pgs td/ceph-helpers 2026-02-25T22:29:56.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1450: test_get_osd_id_used_by_pgs: local dir=td/ceph-helpers 2026-02-25T22:29:56.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1452: test_get_osd_id_used_by_pgs: setup td/ceph-helpers 2026-02-25T22:29:56.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:29:56.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:29:56.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:29:56.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:29:56.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:29:56.634 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:29:56.634 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:29:56.635 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:29:56.635 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:29:56.635 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:29:56.636 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:29:56.636 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:29:56.637 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:29:56.637 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:29:56.638 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:29:56.638 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:29:56.638 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:29:56.639 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:56.639 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:29:56.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:29:56.640 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:29:56.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:29:56.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:29:56.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:29:56.644 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:29:56.644 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:56.644 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:56.644 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:29:56.645 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:29:56.645 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:29:56.645 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:29:56.646 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:29:56.646 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:56.647 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:56.647 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:29:56.648 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:29:56.649 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:29:56.649 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:29:56.649 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:29:56.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1453: test_get_osd_id_used_by_pgs: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:56.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:29:56.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:29:56.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:29:56.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:29:56.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:29:56.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:56.671 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:29:56.671 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:56.671 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:56.671 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:56.671 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:56.671 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:56.672 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:56.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:29:56.699 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:29:56.700 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:29:56.700 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:56.700 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:56.701 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:29:56.702 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:29:56.702 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:56.702 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:56.702 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:56.702 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:56.702 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:56.702 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:56.702 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:56.702 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:56.703 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:29:56.758 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:29:56.758 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:29:56.759 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:29:56.759 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:29:56.759 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:29:56.760 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:29:56.760 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:29:56.760 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:29:56.760 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:29:56.760 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:56.760 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:56.760 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:29:56.761 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:29:56.761 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:29:56.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1454: test_get_osd_id_used_by_pgs: run_mgr td/ceph-helpers x 2026-02-25T22:29:56.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:29:56.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:29:56.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:29:56.817 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:29:56.817 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:29:56.817 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:29:56.932 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:29:56.932 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:56.933 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:56.933 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:56.933 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:56.934 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:56.934 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:56.934 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:56.935 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:29:56.955 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1455: test_get_osd_id_used_by_pgs: run_osd td/ceph-helpers 0 2026-02-25T22:29:56.955 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:29:56.955 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:29:56.955 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:29:56.955 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:29:56.955 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:29:56.955 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:29:56.955 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:29:56.956 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:29:56.956 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:29:56.956 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:29:56.956 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:29:56.956 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:29:56.956 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:29:56.956 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:29:56.956 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:29:56.956 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:29:56.956 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:29:56.956 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:29:56.956 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:29:56.957 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:29:56.957 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:56.957 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:29:56.957 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:29:56.957 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:29:56.957 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:29:56.957 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:29:56.957 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:29:56.957 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:29:56.957 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:29:56.957 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:29:56.957 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:29:56.958 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:29:56.958 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:29:56.958 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:29:56.958 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:29:56.958 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=7b4da0e1-5b5b-44fb-ad61-e12c4dad1d10 2026-02-25T22:29:56.959 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 7b4da0e1-5b5b-44fb-ad61-e12c4dad1d10' 2026-02-25T22:29:56.959 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 7b4da0e1-5b5b-44fb-ad61-e12c4dad1d10 2026-02-25T22:29:56.959 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:29:56.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDkd59phgmaORAAOOXNMAwGMvMFG8FuDyTiFQ== 2026-02-25T22:29:56.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDkd59phgmaORAAOOXNMAwGMvMFG8FuDyTiFQ=="}' 2026-02-25T22:29:56.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 7b4da0e1-5b5b-44fb-ad61-e12c4dad1d10 -i td/ceph-helpers/0/new.json 2026-02-25T22:29:57.083 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:57.091 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:29:57.092 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDkd59phgmaORAAOOXNMAwGMvMFG8FuDyTiFQ== --osd-uuid 7b4da0e1-5b5b-44fb-ad61-e12c4dad1d10 2026-02-25T22:29:57.109 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:57.105+0000 7f5cc6cb4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:57.111 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:57.105+0000 7f5cc6cb4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:57.112 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:57.105+0000 7f5cc6cb4a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:57.112 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:57.109+0000 7f5cc6cb4a40 -1 bdev(0x55ab5681a400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:29:57.112 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:57.109+0000 7f5cc6cb4a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:29:57.906 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:29:57.906 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:29:57.907 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:29:57.907 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:29:57.908 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:29:58.029 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:29:58.030 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:29:58.030 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:29:58.030 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:29:58.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:29:58.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:29:58.053 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:58.049+0000 7ff83e2a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:58.055 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:58.049+0000 7ff83e2a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:58.056 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:58.053+0000 7ff83e2a1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:29:58.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:29:58.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:29:58.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:29:58.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:29:58.137 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:29:58.137 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:29:58.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:58.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:29:58.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:58.138 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:58.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:58.610 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:58.605+0000 7ff83e2a1a40 -1 Falling back to public interface 2026-02-25T22:29:59.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:29:59.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:29:59.264 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:29:59.264 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:29:59.264 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:29:59.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:29:59.488 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:29:59.575 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:29:59.569+0000 7ff83e2a1a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:30:00.489 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:00.489 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:00.490 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:30:00.490 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:30:00.490 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:00.490 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:00.703 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1815041379,v1:127.0.0.1:6801/1815041379] [v2:127.0.0.1:6802/1815041379,v1:127.0.0.1:6803/1815041379] exists,up 7b4da0e1-5b5b-44fb-ad61-e12c4dad1d10 2026-02-25T22:30:00.704 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:00.704 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:00.704 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:00.704 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1456: test_get_osd_id_used_by_pgs: create_rbd_pool 2026-02-25T22:30:00.704 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:30:00.910 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:30:00.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:30:00.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:30:01.200 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:30:01.210 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:30:02.211 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:30:02.490 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1457: test_get_osd_id_used_by_pgs: wait_for_clean 2026-02-25T22:30:02.491 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:30:02.491 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:30:02.491 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:30:02.491 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:30:02.491 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:02.491 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:02.492 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:02.492 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:02.492 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:02.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:30:02.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:30:02.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:30:02.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:30:02.599 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:30:02.599 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:30:02.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:30:02.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:30:02.815 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:02.816 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:30:02.897 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:30:02.897 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:30:02.898 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:30:02.898 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:02.898 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:30:02.898 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:02.899 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:30:02.899 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:30:02.900 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:02.901 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:30:02.901 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:30:02.901 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:30:02.902 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:03.115 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:30:03.115 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:30:04.116 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:30:04.117 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:04.336 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:30:04.336 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:30:04.337 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:30:04.337 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:30:04.625 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:30:04.625 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:30:04.625 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:30:04.625 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:30:04.625 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:30:04.625 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:30:04.626 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:30:04.626 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:30:04.837 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:30:04.837 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:30:04.838 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:30:04.838 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:30:05.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:30:05.126 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:30:05.126 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:30:05.126 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: get_osd_id_used_by_pgs 2026-02-25T22:30:05.127 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: uniq 2026-02-25T22:30:05.127 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: ceph --format json pg dump pgs 2026-02-25T22:30:05.127 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: jq '.pg_stats | .[] | .up[], .acting[]' 2026-02-25T22:30:05.127 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1446: get_osd_id_used_by_pgs: sort 2026-02-25T22:30:05.344 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1458: test_get_osd_id_used_by_pgs: local osd_ids=0 2026-02-25T22:30:05.344 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1459: test_get_osd_id_used_by_pgs: test 0 = 0 2026-02-25T22:30:05.344 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1460: test_get_osd_id_used_by_pgs: teardown td/ceph-helpers 2026-02-25T22:30:05.344 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:30:05.344 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:05.344 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:30:05.344 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:05.345 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:05.345 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:05.345 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:05.345 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:05.452 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:05.452 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:05.453 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:05.453 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:05.455 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:05.455 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:05.455 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:05.456 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:05.456 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:05.457 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:05.457 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:05.458 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:05.459 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:05.459 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:30:05.465 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:05.465 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:05.465 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:05.465 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:30:05.467 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:05.467 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:05.467 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:30:05.468 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_osds td/ceph-helpers 2026-02-25T22:30:05.468 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1022: test_get_osds: local dir=td/ceph-helpers 2026-02-25T22:30:05.468 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1024: test_get_osds: setup td/ceph-helpers 2026-02-25T22:30:05.468 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:30:05.468 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:30:05.468 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:30:05.468 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:05.468 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:30:05.468 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:05.468 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:05.468 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:05.468 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:05.469 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:05.469 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:05.470 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:05.470 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:05.471 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:05.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:05.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:05.472 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:05.473 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:05.473 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:05.474 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:05.474 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:05.475 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:05.476 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:05.476 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:30:05.477 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:05.478 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:05.478 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:05.478 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:30:05.479 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:05.479 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:05.479 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:30:05.480 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:30:05.480 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:05.480 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:05.480 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:30:05.482 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:30:05.483 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:30:05.483 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:30:05.483 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:30:05.483 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1025: test_get_osds: run_mon td/ceph-helpers a --osd_pool_default_size=2 2026-02-25T22:30:05.483 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:30:05.483 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:30:05.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:30:05.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:30:05.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:30:05.484 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=2 2026-02-25T22:30:05.504 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:30:05.504 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:05.505 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:05.505 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:05.506 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:05.506 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:05.506 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:05.506 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=2 2026-02-25T22:30:05.535 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:30:05.536 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:30:05.536 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:05.536 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:05.536 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:30:05.537 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:30:05.537 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:05.537 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:05.537 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:05.538 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:05.538 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:05.538 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:05.538 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:30:05.538 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:05.538 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:30:05.593 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:30:05.593 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:05.593 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:05.593 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:30:05.593 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:30:05.594 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:05.595 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:05.595 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:05.595 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:05.595 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:05.595 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:05.595 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:30:05.595 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:05.595 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:30:05.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1026: test_get_osds: run_mgr td/ceph-helpers x 2026-02-25T22:30:05.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:30:05.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:30:05.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:30:05.649 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:30:05.649 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:30:05.649 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:30:05.763 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:30:05.763 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:05.763 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:05.763 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:05.763 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:05.763 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:05.764 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:05.764 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:05.764 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:05.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1027: test_get_osds: run_osd td/ceph-helpers 0 2026-02-25T22:30:05.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:30:05.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:05.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:30:05.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:05.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:30:05.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:30:05.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:05.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:05.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:05.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:30:05.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:30:05.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:05.786 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:05.787 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:30:05.787 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:05.787 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:05.787 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:05.787 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:05.787 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:05.787 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:05.787 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:05.787 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:05.787 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:05.787 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:05.787 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:05.787 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:30:05.788 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:30:05.788 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:05.788 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:05.788 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:05.788 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:05.788 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:05.788 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:05.788 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:30:05.789 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:05.789 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2074bdd9-4440-4819-a370-8a15413a5cbe 2026-02-25T22:30:05.789 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2074bdd9-4440-4819-a370-8a15413a5cbe' 2026-02-25T22:30:05.789 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 2074bdd9-4440-4819-a370-8a15413a5cbe 2026-02-25T22:30:05.790 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:05.798 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDtd59p7X5mLxAA5jAqJSDYIawDP4QIVlrDWg== 2026-02-25T22:30:05.799 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDtd59p7X5mLxAA5jAqJSDYIawDP4QIVlrDWg=="}' 2026-02-25T22:30:05.799 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2074bdd9-4440-4819-a370-8a15413a5cbe -i td/ceph-helpers/0/new.json 2026-02-25T22:30:05.912 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:30:05.921 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:30:05.922 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDtd59p7X5mLxAA5jAqJSDYIawDP4QIVlrDWg== --osd-uuid 2074bdd9-4440-4819-a370-8a15413a5cbe 2026-02-25T22:30:05.938 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:05.933+0000 7f00bfa13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:05.940 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:05.933+0000 7f00bfa13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:05.941 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:05.937+0000 7f00bfa13a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:05.941 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:05.937+0000 7f00bfa13a40 -1 bdev(0x5604a8ae8400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:05.941 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:05.937+0000 7f00bfa13a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:30:06.611 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:30:06.611 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:06.612 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:30:06.612 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:30:06.613 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:06.731 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:30:06.732 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:30:06.732 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:06.733 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:06.733 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:06.733 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:06.746 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:06.741+0000 7f52f2262a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:06.748 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:06.741+0000 7f52f2262a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:06.749 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:06.745+0000 7f52f2262a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:06.849 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:30:06.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:06.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:30:06.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:06.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:06.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:06.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:06.850 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:30:06.851 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:06.851 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:06.966 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:07.258 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:07.253+0000 7f52f2262a40 -1 Falling back to public interface 2026-02-25T22:30:07.967 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:07.967 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:07.968 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:30:07.968 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:30:07.968 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:07.968 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:08.080 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:08.246 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:08.241+0000 7f52f2262a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:30:09.081 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:09.081 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:09.082 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:30:09.082 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:30:09.082 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:09.083 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:09.300 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3727337262,v1:127.0.0.1:6801/3727337262] [v2:127.0.0.1:6802/3727337262,v1:127.0.0.1:6803/3727337262] exists,up 2074bdd9-4440-4819-a370-8a15413a5cbe 2026-02-25T22:30:09.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:09.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:09.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:09.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1028: test_get_osds: run_osd td/ceph-helpers 1 2026-02-25T22:30:09.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:30:09.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:09.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-25T22:30:09.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:09.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/1 2026-02-25T22:30:09.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:30:09.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:09.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:09.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:09.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/1' 2026-02-25T22:30:09.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/1/journal' 2026-02-25T22:30:09.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:09.304 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:09.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:30:09.305 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:09.305 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:09.305 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:09.305 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:09.305 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:09.305 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:09.305 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:09.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:09.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:09.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:09.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:09.305 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:30:09.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:30:09.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:09.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:09.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:09.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:09.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:09.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:09.306 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/1 2026-02-25T22:30:09.306 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:09.307 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=de05dca0-099e-45e1-aeeb-8d40c909cb0a 2026-02-25T22:30:09.307 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 de05dca0-099e-45e1-aeeb-8d40c909cb0a' 2026-02-25T22:30:09.307 INFO:tasks.workunit.client.0.trial188.stdout:add osd1 de05dca0-099e-45e1-aeeb-8d40c909cb0a 2026-02-25T22:30:09.307 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:09.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDxd59prFyjEhAAZa639X6jCFhwr1px5auo3g== 2026-02-25T22:30:09.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDxd59prFyjEhAAZa639X6jCFhwr1px5auo3g=="}' 2026-02-25T22:30:09.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new de05dca0-099e-45e1-aeeb-8d40c909cb0a -i td/ceph-helpers/1/new.json 2026-02-25T22:30:09.526 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:30:09.535 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/1/new.json 2026-02-25T22:30:09.536 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDxd59prFyjEhAAZa639X6jCFhwr1px5auo3g== --osd-uuid de05dca0-099e-45e1-aeeb-8d40c909cb0a 2026-02-25T22:30:09.553 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:09.549+0000 7ff6fa04aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:09.554 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:09.549+0000 7ff6fa04aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:09.556 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:09.549+0000 7ff6fa04aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:09.556 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:09.553+0000 7ff6fa04aa40 -1 bdev(0x565382a94400 td/ceph-helpers/1/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:09.556 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:09.553+0000 7ff6fa04aa40 -1 bluestore(td/ceph-helpers/1) _read_fsid unparsable uuid 2026-02-25T22:30:10.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/1/keyring 2026-02-25T22:30:10.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:10.248 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-25T22:30:10.248 INFO:tasks.workunit.client.0.trial188.stdout:adding osd1 key to auth repository 2026-02-25T22:30:10.248 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:10.535 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-25T22:30:10.536 INFO:tasks.workunit.client.0.trial188.stdout:start osd.1 2026-02-25T22:30:10.536 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/1 --osd-journal=td/ceph-helpers/1/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:10.536 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:10.536 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:10.537 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:10.559 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:10.553+0000 7fbd486a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:10.561 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:10.557+0000 7fbd486a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:10.562 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:10.557+0000 7fbd486a5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:10.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-25T22:30:10.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:10.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-25T22:30:10.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:10.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:10.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:10.756 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:30:10.757 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:10.757 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:10.757 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:30:10.922 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:10.917+0000 7fbd486a5a40 -1 Falling back to public interface 2026-02-25T22:30:10.973 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:11.525 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:11.521+0000 7fbd486a5a40 -1 osd.1 0 log_to_monitors true 2026-02-25T22:30:11.974 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:11.974 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:11.975 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:30:11.975 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:30:11.975 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:11.975 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-25T22:30:12.194 INFO:tasks.workunit.client.0.trial188.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/76804680,v1:127.0.0.1:6811/76804680] [v2:127.0.0.1:6812/76804680,v1:127.0.0.1:6813/76804680] exists,up de05dca0-099e-45e1-aeeb-8d40c909cb0a 2026-02-25T22:30:12.195 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:12.195 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:12.195 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:12.195 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1029: test_get_osds: create_rbd_pool 2026-02-25T22:30:12.195 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:30:12.404 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:30:12.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:30:12.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:30:12.673 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:30:12.683 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:30:13.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:30:13.941 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1030: test_get_osds: wait_for_clean 2026-02-25T22:30:13.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:30:13.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:30:13.942 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:30:13.942 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:30:13.943 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:13.943 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:13.943 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:13.943 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:13.943 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:14.038 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:30:14.038 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:30:14.038 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:30:14.038 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:30:14.038 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:30:14.038 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:30:14.246 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids='0 2026-02-25T22:30:14.246 INFO:tasks.workunit.client.0.trial188.stderr:1' 2026-02-25T22:30:14.246 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:30:14.246 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:14.246 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:30:14.326 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:30:14.326 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:30:14.326 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:30:14.326 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:14.326 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.1 flush_pg_stats 2026-02-25T22:30:14.403 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=38654705667 2026-02-25T22:30:14.404 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 38654705667 2026-02-25T22:30:14.404 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483 1-38654705667' 2026-02-25T22:30:14.404 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:14.404 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:30:14.404 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:14.405 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:30:14.406 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:30:14.406 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:14.407 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:30:14.407 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:30:14.408 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:30:14.408 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:14.632 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:30:14.633 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:30:15.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:30:15.634 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:15.848 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:30:15.849 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:30:16.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:30:16.850 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:17.071 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836484 -lt 21474836483 2026-02-25T22:30:17.071 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:17.071 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 1-38654705667 2026-02-25T22:30:17.071 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:17.072 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=1 2026-02-25T22:30:17.073 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 1-38654705667 2026-02-25T22:30:17.073 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:17.075 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=38654705667 2026-02-25T22:30:17.075 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.1 seq 38654705667 2026-02-25T22:30:17.075 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.1 seq 38654705667' 2026-02-25T22:30:17.075 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 1 2026-02-25T22:30:17.302 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 38654705667 -lt 38654705667 2026-02-25T22:30:17.302 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:30:17.303 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:30:17.303 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:30:17.590 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:30:17.590 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:30:17.591 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:30:17.591 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:30:17.591 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:30:17.591 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:30:17.591 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:30:17.591 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:30:17.807 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:30:17.808 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:30:17.808 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:30:17.808 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:30:18.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:30:18.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:30:18.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:30:18.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1031: test_get_osds: create_rbd_pool 2026-02-25T22:30:18.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:30:18.306 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:30:18.315 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:30:18.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:30:18.596 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:30:18.607 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:30:19.609 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:30:19.891 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: test_get_osds: get_osds rbd GROUP 2026-02-25T22:30:19.891 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1012: get_osds: local poolname=rbd 2026-02-25T22:30:19.891 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1013: get_osds: local objectname=GROUP 2026-02-25T22:30:19.891 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1032: test_get_osds: grep --quiet '^[0-1] [0-1]$' 2026-02-25T22:30:19.892 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: ceph --format json osd map rbd GROUP 2026-02-25T22:30:19.892 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: jq '.acting | .[]' 2026-02-25T22:30:20.104 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1016: get_osds: local 'osds=1 2026-02-25T22:30:20.104 INFO:tasks.workunit.client.0.trial188.stderr:0' 2026-02-25T22:30:20.104 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1018: get_osds: echo 1 0 2026-02-25T22:30:20.104 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1033: test_get_osds: teardown td/ceph-helpers 2026-02-25T22:30:20.104 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:30:20.104 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:20.105 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:30:20.105 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:20.105 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:20.106 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:20.106 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:20.106 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:20.213 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:20.214 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:20.214 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:20.215 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:20.216 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:20.216 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:20.216 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:20.217 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:20.218 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:20.218 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:20.218 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:20.219 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:20.221 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:20.221 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:30:20.229 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:20.229 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:20.229 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:20.229 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:30:20.231 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:20.231 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:20.231 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:30:20.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_pg td/ceph-helpers 2026-02-25T22:30:20.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1091: test_get_pg: local dir=td/ceph-helpers 2026-02-25T22:30:20.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1093: test_get_pg: setup td/ceph-helpers 2026-02-25T22:30:20.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:30:20.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:30:20.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:30:20.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:20.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:30:20.232 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:20.232 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:20.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:20.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:20.232 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:20.233 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:20.234 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:20.234 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:20.234 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:20.235 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:20.235 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:20.236 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:20.237 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:20.237 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:20.237 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:20.238 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:20.238 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:20.240 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:20.240 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:30:20.241 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:20.241 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:20.241 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:20.241 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:30:20.242 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:20.242 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:20.242 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:30:20.243 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:30:20.244 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:20.244 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:20.244 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:30:20.245 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:30:20.246 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:30:20.246 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:30:20.246 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:30:20.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1094: test_get_pg: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:20.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:30:20.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:30:20.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:30:20.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:30:20.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:30:20.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:20.268 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:30:20.269 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:20.269 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:20.269 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:20.270 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:20.270 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:20.270 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:20.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:20.295 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:30:20.296 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:30:20.296 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:20.296 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:20.296 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:30:20.297 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:30:20.298 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:20.298 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:20.298 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:20.298 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:20.298 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:20.298 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:20.298 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:30:20.298 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:20.298 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:30:20.354 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:30:20.354 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:20.354 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:20.354 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:30:20.355 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:30:20.356 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:20.356 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:20.356 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:20.356 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:20.356 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:20.356 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:20.356 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:30:20.356 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:20.356 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:30:20.411 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1095: test_get_pg: run_mgr td/ceph-helpers x 2026-02-25T22:30:20.411 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:30:20.411 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:30:20.412 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:30:20.412 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:30:20.412 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:30:20.412 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:30:20.524 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:30:20.524 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:20.524 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:20.524 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:20.524 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:20.524 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:20.525 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:20.525 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:20.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:20.546 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1096: test_get_pg: run_osd td/ceph-helpers 0 2026-02-25T22:30:20.546 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:30:20.546 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:20.546 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:30:20.546 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:20.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:30:20.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:30:20.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:20.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:20.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:20.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:30:20.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:30:20.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:20.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:20.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:30:20.547 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:20.547 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:20.548 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:20.548 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:20.548 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:20.548 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:20.548 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:20.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:20.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:20.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:20.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:20.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:30:20.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:30:20.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:20.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:20.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:20.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:20.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:20.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:20.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:30:20.549 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:20.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=2e255770-a5be-4999-829f-65e68c3955f5 2026-02-25T22:30:20.550 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 2e255770-a5be-4999-829f-65e68c3955f5 2026-02-25T22:30:20.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 2e255770-a5be-4999-829f-65e68c3955f5' 2026-02-25T22:30:20.550 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:20.559 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQD8d59pGC8XIRAAHBOnibE40I3x7UID5lrq0A== 2026-02-25T22:30:20.559 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQD8d59pGC8XIRAAHBOnibE40I3x7UID5lrq0A=="}' 2026-02-25T22:30:20.559 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 2e255770-a5be-4999-829f-65e68c3955f5 -i td/ceph-helpers/0/new.json 2026-02-25T22:30:20.668 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:30:20.676 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:30:20.677 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQD8d59pGC8XIRAAHBOnibE40I3x7UID5lrq0A== --osd-uuid 2e255770-a5be-4999-829f-65e68c3955f5 2026-02-25T22:30:20.693 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:20.688+0000 7f72ba1ada40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:20.694 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:20.688+0000 7f72ba1ada40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:20.695 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:20.688+0000 7f72ba1ada40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:20.696 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:20.688+0000 7f72ba1ada40 -1 bdev(0x558090882400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:20.696 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:20.688+0000 7f72ba1ada40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:30:21.424 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:30:21.424 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:21.425 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:30:21.425 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:30:21.426 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:21.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:30:21.550 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:30:21.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:21.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:21.551 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:21.551 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:21.572 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:21.564+0000 7fca320a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:21.574 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:21.568+0000 7fca320a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:21.575 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:21.568+0000 7fca320a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:21.665 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:30:21.665 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:21.665 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:30:21.665 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:21.665 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:21.666 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:21.666 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:21.666 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:30:21.666 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:21.666 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:21.774 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:22.022 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:22.016+0000 7fca320a3a40 -1 Falling back to public interface 2026-02-25T22:30:22.641 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:22.636+0000 7fca320a3a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:30:22.775 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:22.775 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:22.776 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:30:22.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:30:22.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:22.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:22.886 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:23.888 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:23.888 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:23.888 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:30:23.889 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:30:23.889 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:23.889 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:24.105 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/341949877,v1:127.0.0.1:6801/341949877] [v2:127.0.0.1:6802/341949877,v1:127.0.0.1:6803/341949877] exists,up 2e255770-a5be-4999-829f-65e68c3955f5 2026-02-25T22:30:24.106 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:24.106 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:24.106 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:24.106 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1097: test_get_pg: create_rbd_pool 2026-02-25T22:30:24.107 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:30:24.313 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:30:24.322 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:30:24.323 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:30:24.601 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:30:24.611 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:30:25.612 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:30:25.908 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1098: test_get_pg: wait_for_clean 2026-02-25T22:30:25.908 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:30:25.908 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:30:25.908 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:30:25.908 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:30:25.909 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:25.909 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:25.909 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:25.910 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:25.910 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:26.008 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:30:26.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:30:26.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:30:26.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:30:26.009 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:30:26.009 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:30:26.226 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:30:26.226 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:30:26.226 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:26.227 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:30:26.310 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:30:26.310 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:30:26.310 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:30:26.310 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:26.311 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:30:26.311 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:26.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:30:26.312 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:30:26.313 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:26.314 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:30:26.314 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:30:26.314 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:30:26.314 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:26.534 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:30:26.535 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:30:27.536 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:30:27.536 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:27.749 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:30:27.750 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:30:27.750 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:30:27.750 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:30:28.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:30:28.035 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:30:28.036 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:30:28.036 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:30:28.036 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:30:28.036 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:30:28.036 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:30:28.036 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:30:28.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:30:28.248 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:30:28.248 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:30:28.249 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:30:28.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:30:28.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:30:28.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:30:28.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: test_get_pg: get_pg rbd GROUP 2026-02-25T22:30:28.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1084: get_pg: local poolname=rbd 2026-02-25T22:30:28.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1085: get_pg: local objectname=GROUP 2026-02-25T22:30:28.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1099: test_get_pg: grep --quiet '^[0-9]\.[0-9a-f][0-9a-f]*$' 2026-02-25T22:30:28.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: ceph --format json osd map rbd GROUP 2026-02-25T22:30:28.528 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1087: get_pg: jq -r .pgid 2026-02-25T22:30:28.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1100: test_get_pg: teardown td/ceph-helpers 2026-02-25T22:30:28.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:30:28.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:28.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:30:28.757 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:28.757 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:28.757 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:28.757 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:28.757 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:28.865 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:28.866 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:28.866 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:28.867 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:28.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:28.868 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:28.868 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:28.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:28.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:28.869 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:28.870 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:28.871 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:28.872 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:28.872 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:30:28.877 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:28.877 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:28.877 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:28.877 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:30:28.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:28.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:28.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:30:28.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_primary td/ceph-helpers 2026-02-25T22:30:28.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1195: test_get_primary: local dir=td/ceph-helpers 2026-02-25T22:30:28.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1197: test_get_primary: setup td/ceph-helpers 2026-02-25T22:30:28.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:30:28.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:30:28.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:30:28.880 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:28.881 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:30:28.881 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:28.881 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:28.881 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:28.881 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:28.881 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:28.881 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:28.882 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:28.882 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:28.883 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:28.883 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:28.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:28.884 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:28.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:28.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:28.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:28.886 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:28.887 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:28.887 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:28.888 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:30:28.889 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:28.889 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:28.889 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:28.889 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:30:28.890 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:28.890 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:28.890 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:30:28.892 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:30:28.892 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:28.892 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:28.892 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:30:28.893 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:30:28.894 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:30:28.894 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:30:28.894 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:30:28.895 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1198: test_get_primary: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:28.895 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:30:28.895 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:30:28.895 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:30:28.895 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:30:28.895 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:30:28.895 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:28.916 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:30:28.916 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:28.916 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:28.916 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:28.916 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:28.916 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:28.917 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:28.917 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:28.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:30:28.945 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:30:28.945 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:28.945 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:28.946 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:30:28.947 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:30:28.947 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:28.947 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:28.947 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:28.947 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:28.947 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:28.947 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:28.947 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:30:28.947 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:28.948 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:30:29.002 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:30:29.002 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:29.002 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:29.002 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:30:29.003 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:30:29.004 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:29.004 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:29.004 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:29.004 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:29.004 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:29.004 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:29.004 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:30:29.005 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:29.005 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:30:29.058 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1199: test_get_primary: local osd=0 2026-02-25T22:30:29.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1200: test_get_primary: run_mgr td/ceph-helpers x 2026-02-25T22:30:29.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:30:29.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:30:29.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:30:29.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:30:29.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:30:29.059 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:30:29.174 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:30:29.175 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:29.175 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:29.175 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:29.176 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:29.176 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:29.176 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:29.176 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:29.176 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:29.197 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1201: test_get_primary: run_osd td/ceph-helpers 0 2026-02-25T22:30:29.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:30:29.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:29.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:30:29.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:29.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:30:29.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:30:29.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:29.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:29.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:29.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:30:29.198 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:30:29.199 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:29.199 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:29.199 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:30:29.199 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:29.199 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:29.199 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:29.199 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:29.199 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:29.199 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:29.199 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:29.199 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:29.199 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:29.199 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:29.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:29.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:30:29.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:30:29.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:29.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:29.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:29.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:29.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:29.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:29.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:30:29.201 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:29.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=28bf0212-0ef9-4856-a0b9-3d59846eae6d 2026-02-25T22:30:29.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 28bf0212-0ef9-4856-a0b9-3d59846eae6d' 2026-02-25T22:30:29.201 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 28bf0212-0ef9-4856-a0b9-3d59846eae6d 2026-02-25T22:30:29.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:29.211 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAFeJ9p52BbDBAAZikz4rt0yRsvwSKV4tKgCw== 2026-02-25T22:30:29.211 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAFeJ9p52BbDBAAZikz4rt0yRsvwSKV4tKgCw=="}' 2026-02-25T22:30:29.211 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 28bf0212-0ef9-4856-a0b9-3d59846eae6d -i td/ceph-helpers/0/new.json 2026-02-25T22:30:29.326 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:30:29.334 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:30:29.335 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAFeJ9p52BbDBAAZikz4rt0yRsvwSKV4tKgCw== --osd-uuid 28bf0212-0ef9-4856-a0b9-3d59846eae6d 2026-02-25T22:30:29.351 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:29.344+0000 7f5e83b34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:29.352 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:29.348+0000 7f5e83b34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:29.353 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:29.348+0000 7f5e83b34a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:29.354 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:29.348+0000 7f5e83b34a40 -1 bdev(0x55a4f5ed6400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:29.354 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:29.348+0000 7f5e83b34a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:30:30.261 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:30:30.261 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:30.261 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:30:30.262 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:30:30.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:30.262 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:30:30.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:30:30.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:30.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:30.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:30.264 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:30.274 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:30.268+0000 7f64fce00a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:30.277 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:30.272+0000 7f64fce00a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:30.279 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:30.272+0000 7f64fce00a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:30.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:30:30.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:30.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:30:30.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:30.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:30.362 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:30.363 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:30.363 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:30:30.363 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:30.363 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:30.470 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:30.658 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:30.652+0000 7f64fce00a40 -1 Falling back to public interface 2026-02-25T22:30:31.471 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:31.471 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:31.471 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:30:31.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:30:31.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:31.472 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:31.577 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:31.717 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:31.712+0000 7f64fce00a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:30:32.578 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:32.582 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:32.583 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:30:32.583 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:30:32.583 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:32.583 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:32.792 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/2062241481,v1:127.0.0.1:6801/2062241481] [v2:127.0.0.1:6802/2062241481,v1:127.0.0.1:6803/2062241481] exists,up 28bf0212-0ef9-4856-a0b9-3d59846eae6d 2026-02-25T22:30:32.793 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:32.793 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:32.793 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:32.793 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1202: test_get_primary: create_rbd_pool 2026-02-25T22:30:32.793 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:30:32.995 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:30:33.004 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:30:33.004 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:30:33.239 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:30:33.251 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:30:34.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:30:34.558 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1203: test_get_primary: wait_for_clean 2026-02-25T22:30:34.558 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:30:34.558 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:30:34.558 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:30:34.559 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:30:34.559 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:34.559 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:34.560 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:34.560 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:34.560 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:34.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:30:34.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:30:34.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:30:34.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:30:34.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:30:34.667 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:30:34.890 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:30:34.890 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:30:34.890 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:34.891 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:30:34.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-25T22:30:34.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-25T22:30:34.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836484' 2026-02-25T22:30:34.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:34.971 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836484 2026-02-25T22:30:34.971 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:34.972 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:30:34.972 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-25T22:30:34.973 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:34.974 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836484 2026-02-25T22:30:34.974 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836484 2026-02-25T22:30:34.975 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-25T22:30:34.975 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:35.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-25T22:30:35.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:30:36.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:30:36.201 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:36.411 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-25T22:30:36.412 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:30:36.412 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:30:36.412 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:30:36.704 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:30:36.705 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:30:36.705 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:30:36.705 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:30:36.705 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:30:36.705 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:30:36.705 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:30:36.705 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:30:36.918 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:30:36.919 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:30:36.919 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:30:36.919 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:30:37.206 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:30:37.207 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:30:37.207 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:30:37.207 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: get_primary rbd GROUP 2026-02-25T22:30:37.208 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1187: get_primary: local poolname=rbd 2026-02-25T22:30:37.208 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1188: get_primary: local objectname=GROUP 2026-02-25T22:30:37.208 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1190: get_primary: ceph --format json osd map rbd GROUP 2026-02-25T22:30:37.208 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1191: get_primary: jq .acting_primary 2026-02-25T22:30:37.429 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1204: test_get_primary: test 0 = 0 2026-02-25T22:30:37.429 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1205: test_get_primary: teardown td/ceph-helpers 2026-02-25T22:30:37.429 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:30:37.429 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:37.429 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:30:37.429 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:37.430 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:37.430 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:37.430 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:37.430 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:37.537 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:37.538 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:37.539 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:37.539 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:37.541 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:37.541 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:37.541 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:37.542 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:37.542 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:37.542 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:37.543 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:37.544 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:37.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:37.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:30:37.549 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:37.549 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:37.549 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:37.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:30:37.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:37.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:37.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:30:37.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_get_timeout_delays td/ceph-helpers 2026-02-25T22:30:37.551 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: get_timeout_delays 1 2026-02-25T22:30:37.551 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:37.551 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:37.552 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:37.552 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:37.552 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:37.564 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1621: test_get_timeout_delays: test '1 ' = '1 ' 2026-02-25T22:30:37.565 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: get_timeout_delays 5 2026-02-25T22:30:37.566 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:37.566 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:37.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:37.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:37.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:37.584 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1622: test_get_timeout_delays: test '1 2 2 ' = '1 2 2 ' 2026-02-25T22:30:37.585 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: get_timeout_delays 6 2026-02-25T22:30:37.585 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:37.586 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:37.586 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:37.586 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:37.586 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:37.603 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1623: test_get_timeout_delays: test '1 2 3 ' = '1 2 3 ' 2026-02-25T22:30:37.603 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: get_timeout_delays 7 2026-02-25T22:30:37.604 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:37.604 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:37.605 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:37.605 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:37.605 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:37.635 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1624: test_get_timeout_delays: test '1 2 4 ' = '1 2 4 ' 2026-02-25T22:30:37.636 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: get_timeout_delays 8 2026-02-25T22:30:37.636 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:37.636 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:37.637 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:37.637 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:37.637 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:37.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1625: test_get_timeout_delays: test '1 2 4 1 ' = '1 2 4 1 ' 2026-02-25T22:30:37.671 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: get_timeout_delays 1 .1 2026-02-25T22:30:37.671 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:37.671 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:37.672 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:37.672 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:37.672 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:37.696 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1626: test_get_timeout_delays: test '0.1 0.2 0.4 0.3 ' = '0.1 0.2 0.4 0.3 ' 2026-02-25T22:30:37.703 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: get_timeout_delays 1.5 .1 2026-02-25T22:30:37.703 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:37.703 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:37.703 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:37.703 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:37.703 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:37.732 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1627: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 ' = '0.1 0.2 0.4 0.8 ' 2026-02-25T22:30:37.732 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: get_timeout_delays 5 .1 2026-02-25T22:30:37.733 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:37.733 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:37.733 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:37.733 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:37.733 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:37.784 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1628: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 1.9 ' = '0.1 0.2 0.4 0.8 1.6 1.9 ' 2026-02-25T22:30:37.785 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1629: test_get_timeout_delays: get_timeout_delays 6 .1 2026-02-25T22:30:37.785 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:37.785 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:37.786 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:37.786 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:37.786 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:37.836 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1629: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 2.9 ' = '0.1 0.2 0.4 0.8 1.6 2.9 ' 2026-02-25T22:30:37.837 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1630: test_get_timeout_delays: get_timeout_delays 6.3 .1 2026-02-25T22:30:37.837 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:37.838 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:37.838 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:37.838 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:37.838 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:37.877 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1630: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 ' = '0.1 0.2 0.4 0.8 1.6 3.2 ' 2026-02-25T22:30:37.878 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: test_get_timeout_delays: get_timeout_delays 20 .1 2026-02-25T22:30:37.878 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:37.878 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:37.879 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:37.879 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:37.879 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:37.931 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1631: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 7.3 ' 2026-02-25T22:30:37.932 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1632: test_get_timeout_delays: get_timeout_delays 300 .1 0 2026-02-25T22:30:37.932 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:37.932 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:37.933 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:37.933 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:37.933 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:38.037 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1632: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 12.8 25.6 51.2 102.4 95.3 ' 2026-02-25T22:30:38.037 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: test_get_timeout_delays: get_timeout_delays 300 .1 10 2026-02-25T22:30:38.038 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:38.038 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:38.038 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:38.038 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:38.038 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:38.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1633: test_get_timeout_delays: test '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' = '0.1 0.2 0.4 0.8 1.6 3.2 6.4 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 10 7.3 ' 2026-02-25T22:30:38.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:30:38.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_is_clean td/ceph-helpers 2026-02-25T22:30:38.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1552: test_is_clean: local dir=td/ceph-helpers 2026-02-25T22:30:38.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1554: test_is_clean: setup td/ceph-helpers 2026-02-25T22:30:38.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:30:38.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:30:38.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:30:38.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:38.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:30:38.312 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:38.312 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:38.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:38.313 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:38.313 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:38.314 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:38.314 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:38.315 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:38.315 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:38.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:38.316 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:38.317 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:38.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:38.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:38.318 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:38.318 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:38.319 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:38.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:38.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:30:38.322 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:38.322 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:38.322 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:38.322 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:30:38.323 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:38.323 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:38.323 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:30:38.324 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:30:38.325 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:38.325 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:38.325 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:30:38.326 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:30:38.327 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:30:38.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:30:38.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:30:38.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1555: test_is_clean: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:38.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:30:38.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:30:38.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:30:38.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:30:38.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:30:38.328 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:38.349 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:30:38.349 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:38.349 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:38.349 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:38.349 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:38.349 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:38.350 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:38.350 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:38.380 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:30:38.381 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:30:38.381 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:38.381 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:38.382 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:30:38.383 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:30:38.383 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:38.383 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:38.383 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:38.383 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:38.383 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:38.383 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:38.383 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:30:38.384 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:38.384 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:30:38.437 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:30:38.438 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:38.438 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:38.438 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:30:38.439 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:30:38.439 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:38.439 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:38.439 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:38.439 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:38.439 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:38.439 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:38.439 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:30:38.440 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:38.440 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:30:38.494 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1556: test_is_clean: run_mgr td/ceph-helpers x 2026-02-25T22:30:38.494 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:30:38.494 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:30:38.494 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:30:38.494 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:30:38.494 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:30:38.494 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:30:38.603 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:30:38.604 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:38.604 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:38.604 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:38.604 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:38.605 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:38.605 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:38.605 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:38.606 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:38.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1557: test_is_clean: run_osd td/ceph-helpers 0 2026-02-25T22:30:38.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:30:38.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:38.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:30:38.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:38.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:30:38.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:30:38.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:38.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:38.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:38.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:30:38.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:30:38.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:38.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:38.629 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:30:38.629 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:38.629 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:38.629 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:38.629 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:38.629 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:38.630 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:38.630 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:38.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:38.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:38.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:38.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:38.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:30:38.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:30:38.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:38.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:38.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:38.630 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:38.631 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:38.631 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:38.631 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:30:38.631 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:38.631 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=630222ba-299c-42dc-a8c8-9c9ceb5bedda 2026-02-25T22:30:38.632 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 630222ba-299c-42dc-a8c8-9c9ceb5bedda 2026-02-25T22:30:38.632 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 630222ba-299c-42dc-a8c8-9c9ceb5bedda' 2026-02-25T22:30:38.632 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:38.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAOeJ9p9AMIJhAAYZlDiEtY5R9X1hcz2T+++Q== 2026-02-25T22:30:38.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAOeJ9p9AMIJhAAYZlDiEtY5R9X1hcz2T+++Q=="}' 2026-02-25T22:30:38.642 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 630222ba-299c-42dc-a8c8-9c9ceb5bedda -i td/ceph-helpers/0/new.json 2026-02-25T22:30:38.757 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:30:38.765 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:30:38.766 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAOeJ9p9AMIJhAAYZlDiEtY5R9X1hcz2T+++Q== --osd-uuid 630222ba-299c-42dc-a8c8-9c9ceb5bedda 2026-02-25T22:30:38.782 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:38.776+0000 7f9206b4aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:38.784 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:38.776+0000 7f9206b4aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:38.785 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:38.780+0000 7f9206b4aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:38.785 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:38.780+0000 7f9206b4aa40 -1 bdev(0x562ff7306400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:38.785 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:38.780+0000 7f9206b4aa40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:30:39.466 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:30:39.467 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:39.467 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:30:39.468 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:30:39.468 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:39.590 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:30:39.591 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:30:39.591 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:39.591 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:39.591 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:39.591 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:39.614 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:39.608+0000 7f1240e33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:39.617 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:39.612+0000 7f1240e33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:39.618 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:39.612+0000 7f1240e33a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:39.698 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:30:39.698 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:39.699 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:30:39.699 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:39.699 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:39.699 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:39.699 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:30:39.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:39.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:39.700 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:39.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:39.970 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:39.964+0000 7f1240e33a40 -1 Falling back to public interface 2026-02-25T22:30:40.818 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:40.818 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:40.818 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:30:40.818 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:30:40.820 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:40.823 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:40.952 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:41.000 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:40.992+0000 7f1240e33a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:30:41.953 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:41.953 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:41.953 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:30:41.954 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:30:41.954 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:41.954 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:42.167 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/277687479,v1:127.0.0.1:6801/277687479] [v2:127.0.0.1:6802/277687479,v1:127.0.0.1:6803/277687479] exists,up 630222ba-299c-42dc-a8c8-9c9ceb5bedda 2026-02-25T22:30:42.168 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:42.168 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:42.168 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:42.168 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1558: test_is_clean: create_rbd_pool 2026-02-25T22:30:42.169 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:30:42.369 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:30:42.378 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:30:42.378 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:30:42.659 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:30:42.671 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:30:43.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:30:43.989 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1559: test_is_clean: wait_for_clean 2026-02-25T22:30:43.989 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:30:43.989 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:30:43.989 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:30:43.990 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:30:43.990 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:30:43.990 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:30:43.990 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:30:43.991 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:30:43.991 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:30:44.103 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:30:44.103 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:30:44.103 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:30:44.104 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:30:44.104 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:30:44.104 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:30:44.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:30:44.321 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:30:44.321 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:30:44.321 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:30:44.398 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:30:44.398 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:30:44.398 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:30:44.398 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:30:44.399 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:30:44.399 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:30:44.400 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:30:44.400 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:30:44.401 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:30:44.402 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:30:44.402 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:30:44.402 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:30:44.402 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:44.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:30:44.620 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:30:45.621 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:30:45.621 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:30:45.839 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:30:45.839 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:30:45.840 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:30:45.840 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:30:46.130 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:30:46.130 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:30:46.131 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:30:46.131 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:30:46.131 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:30:46.131 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:30:46.131 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:30:46.131 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:30:46.345 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:30:46.346 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:30:46.346 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:30:46.346 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:30:46.639 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:30:46.639 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:30:46.639 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:30:46.639 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1560: test_is_clean: is_clean 2026-02-25T22:30:46.639 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1546: is_clean: get_num_pgs 2026-02-25T22:30:46.640 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:30:46.640 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:30:46.928 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1546: is_clean: num_pgs=4 2026-02-25T22:30:46.928 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1547: is_clean: test 4 '!=' 0 2026-02-25T22:30:46.928 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: is_clean: get_num_active_clean 2026-02-25T22:30:46.928 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:30:46.928 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:30:46.928 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:30:46.929 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:30:46.929 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:30:47.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1548: is_clean: test 4 = 4 2026-02-25T22:30:47.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1561: test_is_clean: teardown td/ceph-helpers 2026-02-25T22:30:47.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:30:47.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:47.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:30:47.142 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:47.142 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:47.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:47.142 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:47.143 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:47.251 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:47.251 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:47.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:47.252 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:47.253 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:47.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:47.254 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:47.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:47.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:47.256 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:47.256 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:47.257 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:47.258 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:47.258 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:30:47.262 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:47.262 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:47.262 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:47.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:30:47.264 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:47.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:47.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:30:47.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_kill_daemon td/ceph-helpers 2026-02-25T22:30:47.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:266: test_kill_daemon: local dir=td/ceph-helpers 2026-02-25T22:30:47.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:267: test_kill_daemon: setup td/ceph-helpers 2026-02-25T22:30:47.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:30:47.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:30:47.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:30:47.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:47.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:30:47.265 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:47.265 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:47.266 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:47.266 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:47.266 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:47.266 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:47.267 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:47.268 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:47.268 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:47.269 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:47.269 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:47.270 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:47.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:47.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:47.271 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:47.271 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:47.272 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:47.273 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:47.274 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:30:47.275 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:47.275 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:47.275 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:47.275 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:30:47.276 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:47.276 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:47.277 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:30:47.278 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:30:47.278 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:47.278 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:47.278 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:30:47.280 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:30:47.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:30:47.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:30:47.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:30:47.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:268: test_kill_daemon: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:47.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:30:47.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:30:47.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:30:47.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:30:47.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:30:47.282 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:47.302 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:30:47.302 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:47.302 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:47.303 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:47.303 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:47.303 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:47.303 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:47.303 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:47.332 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:30:47.333 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:30:47.333 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:47.333 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:47.334 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:30:47.335 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:30:47.335 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:47.335 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:47.335 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:47.335 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:47.335 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:47.335 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:47.335 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:30:47.335 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:47.335 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:30:47.391 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:30:47.391 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:47.392 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:47.392 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:30:47.393 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:30:47.393 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:47.393 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:47.393 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:47.393 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:47.393 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:47.393 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:47.393 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:30:47.394 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:47.394 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:30:47.450 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:269: test_kill_daemon: run_mgr td/ceph-helpers x 2026-02-25T22:30:47.450 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:30:47.450 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:30:47.450 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:30:47.450 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:30:47.450 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:30:47.450 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:30:47.564 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:30:47.564 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:47.565 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:47.565 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:47.565 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:47.565 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:47.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:47.566 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:47.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:47.587 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:270: test_kill_daemon: run_osd td/ceph-helpers 0 2026-02-25T22:30:47.587 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:30:47.587 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:47.587 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:30:47.587 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:47.587 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:30:47.587 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:30:47.588 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:47.588 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:47.588 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:47.588 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:30:47.588 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:30:47.588 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:47.588 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:47.588 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:30:47.588 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:47.588 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:47.588 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:47.588 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:47.589 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:47.589 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:47.589 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:47.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:47.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:47.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:47.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:47.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:30:47.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:30:47.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:47.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:47.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:47.589 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:47.590 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:47.590 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:47.590 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:30:47.590 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:47.590 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=20b6d36b-282c-433e-a90e-4022ab438ad0 2026-02-25T22:30:47.591 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 20b6d36b-282c-433e-a90e-4022ab438ad0 2026-02-25T22:30:47.591 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 20b6d36b-282c-433e-a90e-4022ab438ad0' 2026-02-25T22:30:47.591 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:47.600 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAXeJ9p4z2LIxAAcdq51suLFNmrKgWWC7zfXg== 2026-02-25T22:30:47.600 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAXeJ9p4z2LIxAAcdq51suLFNmrKgWWC7zfXg=="}' 2026-02-25T22:30:47.600 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 20b6d36b-282c-433e-a90e-4022ab438ad0 -i td/ceph-helpers/0/new.json 2026-02-25T22:30:47.712 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:30:47.720 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:30:47.721 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAXeJ9p4z2LIxAAcdq51suLFNmrKgWWC7zfXg== --osd-uuid 20b6d36b-282c-433e-a90e-4022ab438ad0 2026-02-25T22:30:47.737 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:47.732+0000 7fea35992a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:47.738 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:47.732+0000 7fea35992a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:47.739 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:47.732+0000 7fea35992a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:47.740 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:47.732+0000 7fea35992a40 -1 bdev(0x562475274400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:47.740 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:47.732+0000 7fea35992a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:30:48.412 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:30:48.413 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:48.413 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:30:48.414 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:30:48.414 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:48.526 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:30:48.526 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:30:48.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:48.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:48.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:48.527 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:48.542 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:48.536+0000 7fa9ed09fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:48.543 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:48.536+0000 7fa9ed09fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:48.544 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:48.536+0000 7fa9ed09fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:48.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:30:48.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:48.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:30:48.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:48.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:48.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:48.640 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:48.641 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:30:48.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:48.641 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:48.751 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:48.946 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:48.940+0000 7fa9ed09fa40 -1 Falling back to public interface 2026-02-25T22:30:49.622 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:49.616+0000 7fa9ed09fa40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:30:49.752 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:49.752 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:49.753 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:30:49.753 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:30:49.753 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:49.753 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:49.899 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:50.901 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:30:50.901 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:50.901 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:30:50.902 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:30:50.902 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:50.902 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:51.124 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1400790068,v1:127.0.0.1:6801/1400790068] [v2:127.0.0.1:6802/1400790068,v1:127.0.0.1:6803/1400790068] exists,up 20b6d36b-282c-433e-a90e-4022ab438ad0 2026-02-25T22:30:51.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:30:51.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:30:51.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:30:51.125 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:272: test_kill_daemon: name_prefix=osd 2026-02-25T22:30:51.125 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: find td/ceph-helpers 2026-02-25T22:30:51.126 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: grep 'osd[^/]*\.pid' 2026-02-25T22:30:51.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:273: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-25T22:30:51.127 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:280: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid 0 1 2026-02-25T22:30:51.127 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/osd.0.pid 2026-02-25T22:30:51.128 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=97046 2026-02-25T22:30:51.128 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=0 2026-02-25T22:30:51.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local delays=1 2026-02-25T22:30:51.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-25T22:30:51.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:30:51.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 97046 2026-02-25T22:30:51.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-25T22:30:51.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-25T22:30:51.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 1 2026-02-25T22:30:52.129 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:30:52.130 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 97046 2026-02-25T22:30:52.130 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-25T22:30:52.130 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-25T22:30:52.130 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0 2026-02-25T22:30:52.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 1 2026-02-25T22:30:52.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:284: test_kill_daemon: kill_daemon td/ceph-helpers/osd.0.pid TERM 2026-02-25T22:30:52.131 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/osd.0.pid 2026-02-25T22:30:52.133 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=97046 2026-02-25T22:30:52.133 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2026-02-25T22:30:52.133 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-25T22:30:52.133 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-25T22:30:52.133 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:30:52.133 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 97046 2026-02-25T22:30:52.133 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-25T22:30:52.133 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-25T22:30:52.133 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2026-02-25T22:30:52.234 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:30:52.234 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 97046 2026-02-25T22:30:52.234 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-25T22:30:52.234 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-25T22:30:52.234 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.2 2026-02-25T22:30:52.435 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:30:52.435 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 97046 2026-02-25T22:30:52.435 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2026-02-25T22:30:52.436 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2026-02-25T22:30:52.436 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2026-02-25T22:30:52.436 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:287: test_kill_daemon: name_prefix=mgr 2026-02-25T22:30:52.436 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: find td/ceph-helpers 2026-02-25T22:30:52.436 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: grep 'mgr[^/]*\.pid' 2026-02-25T22:30:52.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:288: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-25T22:30:52.439 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:292: test_kill_daemon: kill_daemon td/ceph-helpers/mgr.x.pid TERM 2026-02-25T22:30:52.439 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/mgr.x.pid 2026-02-25T22:30:52.440 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=96190 2026-02-25T22:30:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2026-02-25T22:30:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-25T22:30:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-25T22:30:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:30:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 96190 2026-02-25T22:30:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-25T22:30:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-25T22:30:52.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2026-02-25T22:30:52.542 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:30:52.542 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 96190 2026-02-25T22:30:52.542 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2026-02-25T22:30:52.543 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2026-02-25T22:30:52.543 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2026-02-25T22:30:52.543 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:295: test_kill_daemon: name_prefix=mon 2026-02-25T22:30:52.543 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: find td/ceph-helpers 2026-02-25T22:30:52.543 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: grep 'mon[^/]*\.pid' 2026-02-25T22:30:52.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:296: test_kill_daemon: for pidfile in $(find $dir 2>/dev/null | grep $name_prefix'[^/]*\.pid') 2026-02-25T22:30:52.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:300: test_kill_daemon: kill_daemon td/ceph-helpers/mon.a.pid TERM 2026-02-25T22:30:52.545 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: cat td/ceph-helpers/mon.a.pid 2026-02-25T22:30:52.546 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:246: kill_daemon: local pid=96105 2026-02-25T22:30:52.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:247: kill_daemon: local send_signal=TERM 2026-02-25T22:30:52.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:248: kill_daemon: local 'delays=0.1 0.2 1 1 1 2 3 5 5 5 10 10 20 60 60 60 120' 2026-02-25T22:30:52.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:249: kill_daemon: local exit_code=1 2026-02-25T22:30:52.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:30:52.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -TERM 96105 2026-02-25T22:30:52.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:254: kill_daemon: exit_code=1 2026-02-25T22:30:52.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:259: kill_daemon: send_signal=0 2026-02-25T22:30:52.547 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:260: kill_daemon: sleep 0.1 2026-02-25T22:30:52.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:252: kill_daemon: for try in $delays 0 2026-02-25T22:30:52.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:253: kill_daemon: kill -0 96105 2026-02-25T22:30:52.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:256: kill_daemon: exit_code=0 2026-02-25T22:30:52.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:257: kill_daemon: break 2026-02-25T22:30:52.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:262: kill_daemon: return 0 2026-02-25T22:30:52.648 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:301: test_kill_daemon: timeout 5 ceph status 2026-02-25T22:30:57.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:304: test_kill_daemon: teardown td/ceph-helpers 2026-02-25T22:30:57.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:30:57.652 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:57.653 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:30:57.653 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:57.653 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:57.654 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:57.654 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:57.654 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:57.659 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:57.659 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:57.660 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:57.661 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:57.662 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:57.662 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:57.663 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:57.664 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:57.664 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:57.664 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:57.664 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:57.665 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:57.666 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:57.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:30:57.670 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:57.670 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:57.670 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:57.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:30:57.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:57.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:57.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:30:57.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_kill_daemons td/ceph-helpers 2026-02-25T22:30:57.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:357: test_kill_daemons: local dir=td/ceph-helpers 2026-02-25T22:30:57.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:358: test_kill_daemons: setup td/ceph-helpers 2026-02-25T22:30:57.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:30:57.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:30:57.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:30:57.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:30:57.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:30:57.673 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:30:57.673 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:30:57.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:30:57.673 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:30:57.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:30:57.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:30:57.675 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:30:57.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:30:57.676 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:30:57.677 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:30:57.677 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:30:57.678 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:30:57.679 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:57.679 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:30:57.679 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:30:57.679 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:30:57.680 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:30:57.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:30:57.681 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:30:57.683 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:30:57.683 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:57.683 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:57.683 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:30:57.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:30:57.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:30:57.684 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:30:57.686 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:30:57.686 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:57.686 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:57.686 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:30:57.687 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:30:57.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:30:57.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:30:57.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:30:57.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:359: test_kill_daemons: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:57.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:30:57.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:30:57.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:30:57.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:30:57.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:30:57.689 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:57.710 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:30:57.710 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:57.710 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:57.710 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:57.711 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:57.711 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:57.711 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:57.711 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:30:57.739 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:30:57.740 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:30:57.740 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:57.740 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:57.740 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:30:57.742 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:30:57.742 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:57.742 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:57.742 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:57.742 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:57.742 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:57.742 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:57.742 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:30:57.742 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:57.742 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:30:57.796 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:30:57.796 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:30:57.796 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:30:57.796 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:30:57.796 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:30:57.797 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:30:57.797 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:30:57.797 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:30:57.798 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:30:57.798 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:57.798 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:57.798 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:30:57.798 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:30:57.798 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:30:57.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:360: test_kill_daemons: run_mgr td/ceph-helpers x 2026-02-25T22:30:57.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:30:57.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:30:57.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:30:57.850 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:30:57.851 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:30:57.851 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:30:57.961 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:30:57.961 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:57.961 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:57.961 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:57.961 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:57.961 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:57.962 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:57.962 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:57.963 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:30:57.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:361: test_kill_daemons: run_osd td/ceph-helpers 0 2026-02-25T22:30:57.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:30:57.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:30:57.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:30:57.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:30:57.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:30:57.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:30:57.984 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:30:57.984 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:30:57.984 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:30:57.984 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:30:57.984 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:30:57.984 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:30:57.984 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:30:57.984 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:30:57.984 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:30:57.984 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:30:57.984 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:30:57.984 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:30:57.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:30:57.986 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:30:57.986 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:30:57.986 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:30:57.986 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=be159efc-8e6d-4e84-9ab2-4747cd297d6c 2026-02-25T22:30:57.987 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 be159efc-8e6d-4e84-9ab2-4747cd297d6c' 2026-02-25T22:30:57.987 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 be159efc-8e6d-4e84-9ab2-4747cd297d6c 2026-02-25T22:30:57.987 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:30:57.996 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAheJ9psWwmOxAAKU2WZzuxrPK4J7PiLdBhfQ== 2026-02-25T22:30:57.997 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAheJ9psWwmOxAAKU2WZzuxrPK4J7PiLdBhfQ=="}' 2026-02-25T22:30:57.997 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new be159efc-8e6d-4e84-9ab2-4747cd297d6c -i td/ceph-helpers/0/new.json 2026-02-25T22:30:58.111 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:30:58.119 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:30:58.120 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAheJ9psWwmOxAAKU2WZzuxrPK4J7PiLdBhfQ== --osd-uuid be159efc-8e6d-4e84-9ab2-4747cd297d6c 2026-02-25T22:30:58.136 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:58.128+0000 7fac4c1f2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:58.138 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:58.132+0000 7fac4c1f2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:58.139 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:58.132+0000 7fac4c1f2a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:58.139 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:58.132+0000 7fac4c1f2a40 -1 bdev(0x55b1ddb6a400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:30:58.139 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:58.132+0000 7fac4c1f2a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:30:58.812 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:30:58.812 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:30:58.813 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:30:58.813 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:30:58.813 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:30:58.927 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:30:58.928 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:30:58.928 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:30:58.928 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:30:58.928 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:30:58.928 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:30:58.942 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:58.936+0000 7f88006a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:58.944 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:58.936+0000 7f88006a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:58.945 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:58.940+0000 7f88006a3a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:30:59.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:30:59.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:30:59.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:30:59.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:30:59.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:30:59.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:30:59.040 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:30:59.041 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:30:59.041 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:30:59.041 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:30:59.151 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:30:59.478 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:30:59.472+0000 7f88006a3a40 -1 Falling back to public interface 2026-02-25T22:31:00.153 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:00.153 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:00.153 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:00.153 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:31:00.153 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:00.154 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:00.279 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:00.533 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:00.528+0000 7f88006a3a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:31:01.280 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:01.280 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:01.281 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:31:01.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:31:01.281 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:01.282 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:01.491 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/4113066714,v1:127.0.0.1:6801/4113066714] [v2:127.0.0.1:6802/4113066714,v1:127.0.0.1:6803/4113066714] exists,up be159efc-8e6d-4e84-9ab2-4747cd297d6c 2026-02-25T22:31:01.491 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:01.491 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:01.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:01.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:368: test_kill_daemons: kill_daemons td/ceph-helpers 0 osd 1 2026-02-25T22:31:01.492 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:01.492 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:01.492 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:01.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:01.493 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:02.499 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 1 2026-02-25T22:31:02.500 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:372: test_kill_daemons: kill_daemons td/ceph-helpers TERM osd 2026-02-25T22:31:02.501 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:02.501 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:02.501 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:02.501 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:02.501 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:02.809 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:02.809 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:376: test_kill_daemons: kill_daemons td/ceph-helpers TERM mgr 2026-02-25T22:31:02.809 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:02.809 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:02.810 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:02.810 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:02.810 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:02.917 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:02.918 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:380: test_kill_daemons: kill_daemons td/ceph-helpers TERM 2026-02-25T22:31:02.918 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:02.918 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:02.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:02.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:02.919 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:03.227 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:03.227 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:381: test_kill_daemons: timeout 5 ceph status 2026-02-25T22:31:08.235 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:382: test_kill_daemons: teardown td/ceph-helpers 2026-02-25T22:31:08.235 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:31:08.235 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:31:08.235 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:31:08.235 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:08.235 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:08.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:08.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:08.236 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:08.240 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:08.241 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:31:08.242 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:31:08.242 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:31:08.243 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:31:08.243 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:31:08.244 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:31:08.245 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:08.245 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:31:08.245 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:31:08.245 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:08.246 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:31:08.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:31:08.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:31:08.252 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:31:08.252 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.252 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:08.252 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:31:08.254 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:31:08.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:31:08.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:31:08.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_objectstore_tool td/ceph-helpers 2026-02-25T22:31:08.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1298: test_objectstore_tool: local dir=td/ceph-helpers 2026-02-25T22:31:08.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1300: test_objectstore_tool: setup td/ceph-helpers 2026-02-25T22:31:08.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:31:08.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:31:08.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:31:08.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:31:08.255 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:31:08.255 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:08.255 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:08.256 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:08.256 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:08.256 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:08.257 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:08.257 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:31:08.258 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:31:08.258 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:31:08.259 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:31:08.259 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:31:08.259 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:31:08.260 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:08.261 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:31:08.261 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:31:08.261 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:08.262 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:31:08.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:31:08.263 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:31:08.264 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:31:08.264 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.264 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:08.264 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:31:08.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:31:08.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:31:08.265 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:31:08.267 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:31:08.267 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.267 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:08.267 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:31:08.268 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:31:08.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:31:08.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:31:08.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:31:08.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1301: test_objectstore_tool: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:31:08.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:31:08.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:31:08.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:31:08.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:31:08.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:31:08.270 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:31:08.292 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:31:08.292 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:08.292 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:08.292 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:08.292 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.292 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:08.293 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:08.293 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:31:08.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:31:08.321 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:31:08.321 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:31:08.321 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:31:08.321 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:31:08.323 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:31:08.323 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:31:08.323 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:31:08.323 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:31:08.323 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:31:08.323 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.323 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:08.323 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:31:08.323 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:31:08.323 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:31:08.377 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:31:08.377 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:31:08.377 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:31:08.377 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:31:08.378 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:31:08.378 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:31:08.378 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:31:08.379 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:31:08.379 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:31:08.379 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.379 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:08.379 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:31:08.379 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:31:08.379 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:31:08.432 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1302: test_objectstore_tool: local osd=0 2026-02-25T22:31:08.432 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1303: test_objectstore_tool: run_mgr td/ceph-helpers x 2026-02-25T22:31:08.432 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:31:08.432 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:31:08.432 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:31:08.432 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:31:08.432 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:31:08.432 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:31:08.542 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:31:08.542 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:08.542 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:08.542 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:08.542 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.542 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:08.543 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:08.543 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:31:08.544 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:31:08.565 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1304: test_objectstore_tool: run_osd td/ceph-helpers 0 2026-02-25T22:31:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:31:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:31:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:31:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:31:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:31:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:31:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:31:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:31:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:31:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:31:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:31:08.566 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:31:08.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:31:08.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:31:08.567 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:31:08.567 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:08.567 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:08.567 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:08.567 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:08.567 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:08.567 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:08.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:08.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:31:08.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:31:08.567 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:31:08.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:31:08.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:31:08.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:31:08.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:31:08.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:31:08.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:31:08.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:31:08.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:31:08.568 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:31:08.569 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:31:08.569 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=a4ff0349-eb7e-4c11-b6aa-50db6d8917de 2026-02-25T22:31:08.569 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 a4ff0349-eb7e-4c11-b6aa-50db6d8917de' 2026-02-25T22:31:08.569 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 a4ff0349-eb7e-4c11-b6aa-50db6d8917de 2026-02-25T22:31:08.570 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:31:08.579 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAseJ9p3PNFIhAAbtZrOHX17XvxR4vf1uxT+Q== 2026-02-25T22:31:08.579 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAseJ9p3PNFIhAAbtZrOHX17XvxR4vf1uxT+Q=="}' 2026-02-25T22:31:08.579 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new a4ff0349-eb7e-4c11-b6aa-50db6d8917de -i td/ceph-helpers/0/new.json 2026-02-25T22:31:08.695 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:31:08.702 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:31:08.703 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAseJ9p3PNFIhAAbtZrOHX17XvxR4vf1uxT+Q== --osd-uuid a4ff0349-eb7e-4c11-b6aa-50db6d8917de 2026-02-25T22:31:08.719 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:08.712+0000 7effbedc1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:08.721 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:08.712+0000 7effbedc1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:08.722 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:08.716+0000 7effbedc1a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:08.722 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:08.716+0000 7effbedc1a40 -1 bdev(0x56272cf88400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:31:08.722 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:08.716+0000 7effbedc1a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:31:09.440 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:31:09.440 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:31:09.441 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:31:09.441 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:31:09.442 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:31:09.559 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:31:09.560 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:31:09.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:31:09.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:31:09.560 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:31:09.561 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:31:09.583 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:09.576+0000 7fb0f839ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:09.586 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:09.580+0000 7fb0f839ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:09.587 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:09.580+0000 7fb0f839ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:09.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:31:09.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:31:09.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:31:09.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:31:09.670 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:31:09.671 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:09.671 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:31:09.671 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:31:09.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:09.672 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:09.779 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:10.014 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:10.008+0000 7fb0f839ca40 -1 Falling back to public interface 2026-02-25T22:31:10.701 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:10.696+0000 7fb0f839ca40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:31:10.780 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:10.780 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:10.781 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:31:10.781 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:10.782 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:10.785 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:10.921 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:11.922 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:11.922 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:11.922 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:31:11.923 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:31:11.923 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:11.923 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:12.134 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/1700969685,v1:127.0.0.1:6801/1700969685] [v2:127.0.0.1:6802/1700969685,v1:127.0.0.1:6803/1700969685] exists,up a4ff0349-eb7e-4c11-b6aa-50db6d8917de 2026-02-25T22:31:12.135 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:12.135 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:12.135 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:12.135 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1305: test_objectstore_tool: create_rbd_pool 2026-02-25T22:31:12.135 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:31:12.337 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:31:12.346 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:31:12.346 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:31:12.592 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:31:12.602 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:31:13.603 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:31:13.926 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1306: test_objectstore_tool: wait_for_clean 2026-02-25T22:31:13.926 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:31:13.927 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:31:13.927 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:31:13.927 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:31:13.927 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:13.927 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:13.928 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:13.928 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:13.928 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:14.024 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:31:14.025 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:31:14.025 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:31:14.025 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:31:14.025 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:31:14.025 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:14.231 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:31:14.231 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:14.231 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:14.231 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:14.311 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836483 2026-02-25T22:31:14.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836483 2026-02-25T22:31:14.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836483' 2026-02-25T22:31:14.312 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:14.312 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836483 2026-02-25T22:31:14.312 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:14.313 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:31:14.314 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836483 2026-02-25T22:31:14.314 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:14.315 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836483 2026-02-25T22:31:14.315 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836483' 2026-02-25T22:31:14.316 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836483 2026-02-25T22:31:14.316 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:14.541 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836483 2026-02-25T22:31:14.541 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:15.542 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:31:15.543 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:15.756 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836483 -lt 21474836483 2026-02-25T22:31:15.757 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:31:15.757 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:15.758 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:16.043 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:31:16.043 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:31:16.044 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:31:16.044 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:31:16.045 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:31:16.045 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:31:16.045 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:31:16.045 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:31:16.248 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:31:16.248 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:31:16.249 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:16.249 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:16.523 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:31:16.523 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:31:16.523 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:31:16.524 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1307: test_objectstore_tool: rados --pool rbd put GROUP /etc/group 2026-02-25T22:31:16.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1308: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 GROUP get-bytes 2026-02-25T22:31:16.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/ceph-helpers 2026-02-25T22:31:16.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:31:16.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-25T22:31:16.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:31:16.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 GROUP get-bytes 2026-02-25T22:31:16.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/ceph-helpers 2026-02-25T22:31:16.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:31:16.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-25T22:31:16.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:31:16.545 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2026-02-25T22:31:16.546 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1309: test_objectstore_tool: diff - /etc/group 2026-02-25T22:31:16.546 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:16.546 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:16.546 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:16.546 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:16.546 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:16.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:16.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 GROUP get-bytes 2026-02-25T22:31:16.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/ceph-helpers 2026-02-25T22:31:16.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:31:16.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-25T22:31:16.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:31:16.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2026-02-25T22:31:16.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 GROUP get-bytes 2026-02-25T22:31:17.027 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1268: _objectstore_tool_nowait: activate_osd td/ceph-helpers 0 2026-02-25T22:31:17.027 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:837: activate_osd: local dir=td/ceph-helpers 2026-02-25T22:31:17.027 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:838: activate_osd: shift 2026-02-25T22:31:17.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:839: activate_osd: local id=0 2026-02-25T22:31:17.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:840: activate_osd: shift 2026-02-25T22:31:17.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:841: activate_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:31:17.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:843: activate_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:31:17.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:844: activate_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:31:17.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:845: activate_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:31:17.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:846: activate_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:31:17.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:847: activate_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:31:17.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:848: activate_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:31:17.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:849: activate_osd: ceph_args+=' --chdir=' 2026-02-25T22:31:17.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:850: activate_osd: ceph_args+= 2026-02-25T22:31:17.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:851: activate_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:31:17.029 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: get_asok_path 2026-02-25T22:31:17.029 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:17.029 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:17.029 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:17.029 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:17.029 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:17.029 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:17.029 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:852: activate_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:17.029 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:853: activate_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:31:17.029 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:854: activate_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:31:17.029 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:855: activate_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:31:17.029 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:856: activate_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:31:17.030 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:857: activate_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:31:17.030 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:858: activate_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:31:17.030 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:859: activate_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:31:17.030 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:860: activate_osd: ceph_args+=' ' 2026-02-25T22:31:17.030 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:861: activate_osd: ceph_args+= 2026-02-25T22:31:17.030 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:862: activate_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:31:17.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:864: activate_osd: echo start osd.0 2026-02-25T22:31:17.031 INFO:tasks.workunit.client.0.trial188.stderr:start osd.0 2026-02-25T22:31:17.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:865: activate_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:31:17.031 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: cat td/ceph-helpers/0/whoami 2026-02-25T22:31:17.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:867: activate_osd: '[' 0 = 0 ']' 2026-02-25T22:31:17.031 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: ceph osd dump --format=json 2026-02-25T22:31:17.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: jq '.flags_set[]' 2026-02-25T22:31:17.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:870: activate_osd: grep -q '"noup"' 2026-02-25T22:31:17.052 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:17.043+0000 7f182a3d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:17.054 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:17.047+0000 7f182a3d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:17.054 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:17.047+0000 7f182a3d0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:17.249 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:873: activate_osd: wait_for_osd up 0 2026-02-25T22:31:17.249 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:31:17.249 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:31:17.249 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:31:17.250 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:31:17.250 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:17.250 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:31:17.250 INFO:tasks.workunit.client.0.trial188.stderr:0 2026-02-25T22:31:17.250 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:17.250 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:17.455 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:17.447+0000 7f182a3d0a40 -1 Falling back to public interface 2026-02-25T22:31:17.458 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:18.065 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:18.055+0000 7f182a3d0a40 -1 osd.0 12 log_to_monitors true 2026-02-25T22:31:18.459 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:18.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:18.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:18.460 INFO:tasks.workunit.client.0.trial188.stderr:1 2026-02-25T22:31:18.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:18.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:18.674 INFO:tasks.workunit.client.0.trial188.stderr:osd.0 up in weight 1 up_from 15 up_thru 15 down_at 12 last_clean_interval [5,12) [v2:127.0.0.1:6800/2499442296,v1:127.0.0.1:6801/2499442296] [v2:127.0.0.1:6802/2499442296,v1:127.0.0.1:6803/2499442296] exists,up a4ff0349-eb7e-4c11-b6aa-50db6d8917de 2026-02-25T22:31:18.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:18.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:18.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:18.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1294: objectstore_tool: wait_for_clean 2026-02-25T22:31:18.674 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:31:18.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:31:18.675 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:31:18.675 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:31:18.675 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:18.676 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:18.676 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:18.676 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:18.676 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:18.765 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:31:18.765 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:31:18.766 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:31:18.766 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:31:18.766 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:31:18.766 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:18.987 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:31:18.987 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:18.987 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:18.987 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:19.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=64424509443 2026-02-25T22:31:19.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 64424509443 2026-02-25T22:31:19.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-64424509443' 2026-02-25T22:31:19.066 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:19.067 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-64424509443 2026-02-25T22:31:19.067 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:19.068 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:31:19.069 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-64424509443 2026-02-25T22:31:19.069 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:19.070 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=64424509443 2026-02-25T22:31:19.070 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 64424509443' 2026-02-25T22:31:19.070 INFO:tasks.workunit.client.0.trial188.stderr:waiting osd.0 seq 64424509443 2026-02-25T22:31:19.071 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:19.293 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 1 -lt 64424509443 2026-02-25T22:31:19.293 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:20.294 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:31:20.294 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:20.514 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 1 -lt 64424509443 2026-02-25T22:31:20.514 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:21.515 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:31:21.515 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:21.727 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 64424509443 -lt 64424509443 2026-02-25T22:31:21.728 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:31:21.728 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:21.728 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:22.011 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:31:22.011 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:31:22.012 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:31:22.012 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:31:22.012 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:31:22.012 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:31:22.013 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:31:22.013 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:31:22.219 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:31:22.220 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:31:22.220 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:22.220 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:22.506 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:31:22.506 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:31:22.507 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:31:22.508 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1310: test_objectstore_tool: objectstore_tool td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-25T22:31:22.508 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1288: objectstore_tool: local dir=td/ceph-helpers 2026-02-25T22:31:22.508 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1289: objectstore_tool: shift 2026-02-25T22:31:22.508 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1290: objectstore_tool: local id=0 2026-02-25T22:31:22.509 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1291: objectstore_tool: shift 2026-02-25T22:31:22.509 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: _objectstore_tool_nowait td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-25T22:31:22.509 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1260: _objectstore_tool_nowait: local dir=td/ceph-helpers 2026-02-25T22:31:22.509 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1261: _objectstore_tool_nowait: shift 2026-02-25T22:31:22.509 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1262: _objectstore_tool_nowait: local id=0 2026-02-25T22:31:22.509 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1263: _objectstore_tool_nowait: shift 2026-02-25T22:31:22.509 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1265: _objectstore_tool_nowait: kill_daemons td/ceph-helpers TERM osd.0 2026-02-25T22:31:22.509 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:22.509 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:22.509 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:22.509 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:22.509 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:22.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:22.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: _objectstore_tool_nodown td/ceph-helpers 0 NOTEXISTS get-bytes 2026-02-25T22:31:22.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1248: _objectstore_tool_nodown: local dir=td/ceph-helpers 2026-02-25T22:31:22.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1249: _objectstore_tool_nodown: shift 2026-02-25T22:31:22.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1250: _objectstore_tool_nodown: local id=0 2026-02-25T22:31:22.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1251: _objectstore_tool_nodown: shift 2026-02-25T22:31:22.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1252: _objectstore_tool_nodown: local osd_data=td/ceph-helpers/0 2026-02-25T22:31:22.816 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1255: _objectstore_tool_nodown: ceph-objectstore-tool --data-path td/ceph-helpers/0 NOTEXISTS get-bytes 2026-02-25T22:31:22.880 INFO:tasks.workunit.client.0.trial188.stderr:No object id 'NOTEXISTS' found or invalid JSON specified 2026-02-25T22:31:22.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1256: _objectstore_tool_nodown: return 1 2026-02-25T22:31:22.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1267: _objectstore_tool_nowait: return 1 2026-02-25T22:31:22.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1293: objectstore_tool: return 1 2026-02-25T22:31:22.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1311: test_objectstore_tool: teardown td/ceph-helpers 2026-02-25T22:31:22.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:31:22.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:31:22.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:31:22.971 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:22.971 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:22.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:22.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:22.971 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:23.078 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:23.078 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:31:23.079 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:31:23.080 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:31:23.081 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:31:23.081 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:31:23.082 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:31:23.083 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:23.083 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:31:23.083 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:31:23.083 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:23.084 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:31:23.086 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:31:23.086 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:31:23.092 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:31:23.092 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:23.092 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:23.092 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:31:23.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:31:23.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:31:23.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:31:23.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_pg_schedule_scrub td/ceph-helpers 2026-02-25T22:31:23.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1987: test_pg_schedule_scrub: local dir=td/ceph-helpers 2026-02-25T22:31:23.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1989: test_pg_schedule_scrub: setup td/ceph-helpers 2026-02-25T22:31:23.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:31:23.095 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:31:23.096 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:31:23.096 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:31:23.096 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:31:23.096 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:23.096 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:23.096 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:23.096 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:23.096 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:23.097 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:23.097 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:31:23.098 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:31:23.098 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:31:23.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:31:23.099 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:31:23.099 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:31:23.100 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:23.100 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:31:23.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:31:23.101 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:23.102 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:31:23.103 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:31:23.103 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:31:23.104 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:31:23.104 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:23.104 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:23.104 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:31:23.105 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:31:23.105 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:31:23.105 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:31:23.107 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:31:23.107 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:23.107 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:23.107 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:31:23.108 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:31:23.109 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:31:23.110 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:31:23.110 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:31:23.110 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1990: test_pg_schedule_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:31:23.110 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:31:23.110 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:31:23.110 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:31:23.110 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:31:23.110 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:31:23.110 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:31:23.130 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:31:23.130 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:23.130 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:23.130 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:23.130 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:23.131 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:23.131 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:23.131 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:31:23.167 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:31:23.168 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:31:23.168 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:31:23.168 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:31:23.168 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:31:23.169 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:31:23.169 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:31:23.170 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:31:23.170 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:31:23.170 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:31:23.170 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:23.170 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:23.170 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:31:23.170 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:31:23.170 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:31:23.224 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:31:23.224 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:31:23.224 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:31:23.224 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:31:23.225 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:31:23.225 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:31:23.225 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:31:23.225 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:31:23.225 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:31:23.225 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:23.225 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:23.226 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:31:23.226 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:31:23.226 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:31:23.279 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1991: test_pg_schedule_scrub: run_mgr td/ceph-helpers x --mgr_stats_period=1 2026-02-25T22:31:23.279 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:31:23.279 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:31:23.279 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:31:23.279 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:31:23.279 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:31:23.280 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:31:23.390 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:31:23.390 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:23.391 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:23.391 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:23.391 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:23.392 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:23.392 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:23.392 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:31:23.392 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr --mgr_stats_period=1 2026-02-25T22:31:23.413 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1992: test_pg_schedule_scrub: run_osd td/ceph-helpers 0 2026-02-25T22:31:23.413 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:31:23.413 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:31:23.413 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:31:23.413 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:31:23.413 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:31:23.414 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:31:23.414 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:31:23.414 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:31:23.414 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:31:23.414 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:31:23.414 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:31:23.414 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:31:23.414 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:31:23.414 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:31:23.414 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:31:23.414 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:23.414 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:23.414 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:23.415 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:23.415 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:23.415 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:23.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:23.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:31:23.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:31:23.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:31:23.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:31:23.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:31:23.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:31:23.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:31:23.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:31:23.415 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:31:23.416 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:31:23.416 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:31:23.416 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:31:23.416 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:31:23.416 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0e5dd5ed-83ae-4e52-8264-e139abceba34 2026-02-25T22:31:23.416 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 0e5dd5ed-83ae-4e52-8264-e139abceba34' 2026-02-25T22:31:23.417 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 0e5dd5ed-83ae-4e52-8264-e139abceba34 2026-02-25T22:31:23.417 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:31:23.427 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA7eJ9pcBU0GRAA1SHBPGSFRBi6jkHXgdwnsA== 2026-02-25T22:31:23.427 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA7eJ9pcBU0GRAA1SHBPGSFRBi6jkHXgdwnsA=="}' 2026-02-25T22:31:23.427 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0e5dd5ed-83ae-4e52-8264-e139abceba34 -i td/ceph-helpers/0/new.json 2026-02-25T22:31:23.543 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:31:23.551 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:31:23.552 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA7eJ9pcBU0GRAA1SHBPGSFRBi6jkHXgdwnsA== --osd-uuid 0e5dd5ed-83ae-4e52-8264-e139abceba34 2026-02-25T22:31:23.567 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:23.559+0000 7f0639b8da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:23.569 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:23.559+0000 7f0639b8da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:23.570 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:23.563+0000 7f0639b8da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:23.570 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:23.563+0000 7f0639b8da40 -1 bdev(0x5618e04c8400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:31:23.570 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:23.563+0000 7f0639b8da40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:31:24.319 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:31:24.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:31:24.320 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:31:24.321 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:31:24.321 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:31:24.442 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:31:24.443 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:31:24.443 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:31:24.443 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:31:24.443 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:31:24.443 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:31:24.465 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:24.459+0000 7f5132574a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:24.468 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:24.459+0000 7f5132574a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:24.469 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:24.459+0000 7f5132574a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:24.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:31:24.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:31:24.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:31:24.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:31:24.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:31:24.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:24.550 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:31:24.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:31:24.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:24.551 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:24.665 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:24.851 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:24.843+0000 7f5132574a40 -1 Falling back to public interface 2026-02-25T22:31:25.488 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:25.479+0000 7f5132574a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:31:25.666 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:25.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:25.667 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:31:25.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:25.667 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:25.668 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:25.803 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:26.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:26.804 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:26.805 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:31:26.805 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:31:26.805 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:26.806 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:27.014 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/3652829953,v1:127.0.0.1:6801/3652829953] [v2:127.0.0.1:6802/3652829953,v1:127.0.0.1:6803/3652829953] exists,up 0e5dd5ed-83ae-4e52-8264-e139abceba34 2026-02-25T22:31:27.014 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:27.014 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:27.015 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:27.015 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1993: test_pg_schedule_scrub: create_rbd_pool 2026-02-25T22:31:27.015 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:31:27.214 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:31:27.223 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:31:27.223 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:31:27.495 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:31:27.504 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:31:28.505 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:31:28.775 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1994: test_pg_schedule_scrub: wait_for_clean 2026-02-25T22:31:28.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:31:28.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:31:28.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:31:28.777 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:31:28.777 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:28.777 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:28.777 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:28.777 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:28.777 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:28.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:31:28.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:31:28.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:31:28.886 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:31:28.886 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:31:28.886 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:29.109 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:31:29.109 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:29.110 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:29.110 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:29.187 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836485 2026-02-25T22:31:29.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836485 2026-02-25T22:31:29.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836485' 2026-02-25T22:31:29.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:29.188 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836485 2026-02-25T22:31:29.188 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:29.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:31:29.190 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836485 2026-02-25T22:31:29.190 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:29.191 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836485 2026-02-25T22:31:29.191 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836485' 2026-02-25T22:31:29.191 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836485 2026-02-25T22:31:29.192 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:29.419 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836481 -lt 21474836485 2026-02-25T22:31:29.419 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:30.420 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:31:30.421 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:30.631 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836485 -lt 21474836485 2026-02-25T22:31:30.631 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:31:30.632 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:30.632 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:30.903 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:31:30.903 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:31:30.904 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:31:30.904 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:31:30.905 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:31:30.905 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:31:30.905 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:31:30.905 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:31:31.111 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:31:31.111 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:31:31.112 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:31.112 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:31.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:31:31.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:31:31.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:31:31.387 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1995: test_pg_schedule_scrub: pg_schedule_scrub 1.0 2026-02-25T22:31:31.388 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1969: pg_schedule_scrub: local pgid=1.0 2026-02-25T22:31:31.388 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1971: pg_schedule_scrub: wait_for_pg_clean 1.0 2026-02-25T22:31:31.388 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-25T22:31:31.388 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-25T22:31:31.389 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:31.389 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:31.389 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:31.389 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:31.389 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:31.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-25T22:31:31.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-25T22:31:31.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-25T22:31:31.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-25T22:31:31.650 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:31:31.650 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:31.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:31:31.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:31.867 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:31.867 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:31.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836490 2026-02-25T22:31:31.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836490 2026-02-25T22:31:31.944 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836490' 2026-02-25T22:31:31.945 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:31.945 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836490 2026-02-25T22:31:31.945 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:31.946 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:31:31.947 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836490 2026-02-25T22:31:31.947 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:31.948 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836490 2026-02-25T22:31:31.948 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836490 2026-02-25T22:31:31.949 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836490' 2026-02-25T22:31:31.949 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:32.169 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836487 -lt 21474836490 2026-02-25T22:31:32.169 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:33.170 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:31:33.171 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:33.379 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836487 -lt 21474836490 2026-02-25T22:31:33.380 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:34.380 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:31:34.381 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:34.607 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836491 -lt 21474836490 2026-02-25T22:31:34.607 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-25T22:31:34.607 INFO:tasks.workunit.client.0.trial188.stdout:#---------- 1.0 loop 0 2026-02-25T22:31:34.608 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-25T22:31:34.608 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-25T22:31:34.608 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-25T22:31:34.609 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-25T22:31:34.609 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-25T22:31:34.609 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-25T22:31:34.691 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-25T22:31:34.691 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-25T22:31:34.691 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-25T22:31:34.691 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-25T22:31:34.691 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: pg_schedule_scrub: get_last_scrub_stamp 1.0 2026-02-25T22:31:34.691 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:31:34.692 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:31:34.692 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:31:34.692 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:31:34.903 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1972: pg_schedule_scrub: local last_scrub=2026-02-25T22:31:27.439476+0000 2026-02-25T22:31:34.903 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1973: pg_schedule_scrub: ceph tell 1.0 schedule-scrub 2026-02-25T22:31:34.975 INFO:tasks.workunit.client.0.trial188.stdout:{ 2026-02-25T22:31:34.975 INFO:tasks.workunit.client.0.trial188.stdout: "deep": false, 2026-02-25T22:31:34.975 INFO:tasks.workunit.client.0.trial188.stdout: "must": false, 2026-02-25T22:31:34.975 INFO:tasks.workunit.client.0.trial188.stdout: "stamp": "2026-02-23T10:31:34.971396+0000" 2026-02-25T22:31:34.975 INFO:tasks.workunit.client.0.trial188.stdout:} 2026-02-25T22:31:34.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1974: pg_schedule_scrub: wait_for_scrub 1.0 2026-02-25T22:31:27.439476+0000 2026-02-25T22:31:34.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-25T22:31:34.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-25T22:31:27.439476+0000 2026-02-25T22:31:34.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-25T22:31:34.983 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-25T22:31:34.984 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:31:34.984 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:31:34.984 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:31:34.984 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:31:34.984 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:31:34.984 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:31:35.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-25T22:31:27.439476+0000 '>' 2026-02-25T22:31:27.439476+0000 2026-02-25T22:31:35.200 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-25T22:31:36.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-25T22:31:36.202 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:31:36.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:31:36.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:31:36.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:31:36.203 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:31:36.203 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:31:36.416 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-25T22:31:27.439476+0000 '>' 2026-02-25T22:31:27.439476+0000 2026-02-25T22:31:36.416 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-25T22:31:37.417 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-25T22:31:37.417 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:31:37.418 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:31:37.418 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:31:37.418 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:31:37.418 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:31:37.418 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:31:37.627 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-25T22:31:27.439476+0000 '>' 2026-02-25T22:31:27.439476+0000 2026-02-25T22:31:37.627 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-25T22:31:38.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-25T22:31:38.628 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:31:38.629 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:31:38.629 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:31:38.629 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:31:38.629 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:31:38.630 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:31:38.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-23T10:31:34.971396+0000 '>' 2026-02-25T22:31:27.439476+0000 2026-02-25T22:31:38.840 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-25T22:31:39.841 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-25T22:31:39.841 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:31:39.842 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:31:39.842 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:31:39.842 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:31:39.842 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:31:39.843 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:31:40.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-25T22:31:38.308438+0000 '>' 2026-02-25T22:31:27.439476+0000 2026-02-25T22:31:40.053 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-25T22:31:40.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1996: test_pg_schedule_scrub: teardown td/ceph-helpers 2026-02-25T22:31:40.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:31:40.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:31:40.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:31:40.054 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:40.054 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:40.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:40.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:40.054 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:40.162 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:40.162 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:31:40.163 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:31:40.163 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:31:40.165 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:31:40.165 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:31:40.165 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:31:40.166 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:40.166 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:31:40.166 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:31:40.166 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:40.167 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:31:40.168 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:31:40.168 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:31:40.174 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:31:40.174 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:40.174 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:40.175 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:31:40.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:31:40.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:31:40.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2425: run_tests: for func in $funcs 2026-02-25T22:31:40.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2426: run_tests: test_pg_scrub td/ceph-helpers 2026-02-25T22:31:40.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1941: test_pg_scrub: local dir=td/ceph-helpers 2026-02-25T22:31:40.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1943: test_pg_scrub: setup td/ceph-helpers 2026-02-25T22:31:40.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/ceph-helpers 2026-02-25T22:31:40.178 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/ceph-helpers 2026-02-25T22:31:40.178 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-25T22:31:40.178 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-25T22:31:40.178 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-25T22:31:40.178 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:40.178 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:40.178 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:40.178 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:40.178 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:40.179 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:40.179 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-25T22:31:40.180 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-25T22:31:40.180 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-25T22:31:40.182 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-25T22:31:40.182 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-25T22:31:40.182 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-25T22:31:40.183 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:40.183 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-25T22:31:40.184 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-25T22:31:40.184 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-25T22:31:40.185 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-25T22:31:40.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-25T22:31:40.186 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-25T22:31:40.187 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-25T22:31:40.187 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:40.188 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:40.188 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-25T22:31:40.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-25T22:31:40.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-25T22:31:40.189 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/ceph-helpers 2026-02-25T22:31:40.190 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-25T22:31:40.190 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:40.190 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:40.191 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.55572 2026-02-25T22:31:40.192 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-25T22:31:40.193 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-25T22:31:40.193 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-25T22:31:40.193 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/ceph-helpers 1' TERM HUP INT 2026-02-25T22:31:40.193 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1944: test_pg_scrub: run_mon td/ceph-helpers a --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:31:40.194 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/ceph-helpers 2026-02-25T22:31:40.194 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-25T22:31:40.194 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-25T22:31:40.194 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-25T22:31:40.194 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/ceph-helpers/a 2026-02-25T22:31:40.194 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/ceph-helpers/a --run-dir=td/ceph-helpers --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:31:40.215 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-25T22:31:40.215 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:40.215 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:40.216 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:40.216 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:40.216 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:40.216 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:40.217 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/ceph-helpers/a '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --mon-cluster-log-file=td/ceph-helpers/log --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --osd_pool_default_size=1 --mon_allow_pool_size_one=true 2026-02-25T22:31:40.246 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-25T22:31:40.247 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-25T22:31:40.247 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:31:40.247 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:31:40.247 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-25T22:31:40.248 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-25T22:31:40.248 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:31:40.248 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:31:40.249 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:31:40.249 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:31:40.249 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:40.249 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:40.249 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:31:40.249 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:31:40.249 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get fsid 2026-02-25T22:31:40.306 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-25T22:31:40.306 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-25T22:31:40.307 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-25T22:31:40.307 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-25T22:31:40.307 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-25T22:31:40.307 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-25T22:31:40.307 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-25T22:31:40.307 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-25T22:31:40.308 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-25T22:31:40.308 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:40.308 INFO:tasks.workunit.client.0.trial188.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:40.308 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.55572/ceph-mon.a.asok 2026-02-25T22:31:40.308 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-25T22:31:40.308 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.55572/ceph-mon.a.asok config get mon_host 2026-02-25T22:31:40.363 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1945: test_pg_scrub: run_mgr td/ceph-helpers x 2026-02-25T22:31:40.363 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/ceph-helpers 2026-02-25T22:31:40.363 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-25T22:31:40.363 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-25T22:31:40.363 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-25T22:31:40.363 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/ceph-helpers/x 2026-02-25T22:31:40.363 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-25T22:31:40.475 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-25T22:31:40.475 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:40.475 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:40.476 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:40.476 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:40.476 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:40.476 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:40.477 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:31:40.477 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/ceph-helpers/x '--log-file=td/ceph-helpers/$name.log' '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --run-dir=td/ceph-helpers '--pid-file=td/ceph-helpers/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-25T22:31:40.500 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1946: test_pg_scrub: run_osd td/ceph-helpers 0 2026-02-25T22:31:40.501 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/ceph-helpers 2026-02-25T22:31:40.501 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-25T22:31:40.501 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-25T22:31:40.501 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-25T22:31:40.501 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/ceph-helpers/0 2026-02-25T22:31:40.501 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 ' 2026-02-25T22:31:40.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-25T22:31:40.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-25T22:31:40.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-25T22:31:40.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/ceph-helpers/0' 2026-02-25T22:31:40.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/ceph-helpers/0/journal' 2026-02-25T22:31:40.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-25T22:31:40.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-25T22:31:40.502 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/ceph-helpers' 2026-02-25T22:31:40.502 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-25T22:31:40.502 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-25T22:31:40.502 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-25T22:31:40.502 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-25T22:31:40.503 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-25T22:31:40.503 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-25T22:31:40.503 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:40.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' 2026-02-25T22:31:40.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-25T22:31:40.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-25T22:31:40.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-25T22:31:40.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/ceph-helpers/$name.log' 2026-02-25T22:31:40.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/ceph-helpers/$name.pid' 2026-02-25T22:31:40.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-25T22:31:40.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-25T22:31:40.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-25T22:31:40.503 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-25T22:31:40.504 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-25T22:31:40.504 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-25T22:31:40.504 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/ceph-helpers/0 2026-02-25T22:31:40.504 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-25T22:31:40.504 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1c2f7334-02c4-4bac-bff3-9cf34879eeb9 2026-02-25T22:31:40.504 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 1c2f7334-02c4-4bac-bff3-9cf34879eeb9' 2026-02-25T22:31:40.504 INFO:tasks.workunit.client.0.trial188.stdout:add osd0 1c2f7334-02c4-4bac-bff3-9cf34879eeb9 2026-02-25T22:31:40.505 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-25T22:31:40.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBMeJ9p7fEpHhAAFoR16HLhI3uKfzo1jEy1Iw== 2026-02-25T22:31:40.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBMeJ9p7fEpHhAAFoR16HLhI3uKfzo1jEy1Iw=="}' 2026-02-25T22:31:40.511 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1c2f7334-02c4-4bac-bff3-9cf34879eeb9 -i td/ceph-helpers/0/new.json 2026-02-25T22:31:40.627 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:31:40.635 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/ceph-helpers/0/new.json 2026-02-25T22:31:40.636 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBMeJ9p7fEpHhAAFoR16HLhI3uKfzo1jEy1Iw== --osd-uuid 1c2f7334-02c4-4bac-bff3-9cf34879eeb9 2026-02-25T22:31:40.652 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:40.643+0000 7fb9b8656a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:40.654 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:40.647+0000 7fb9b8656a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:40.655 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:40.647+0000 7fb9b8656a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:40.656 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:40.647+0000 7fb9b8656a40 -1 bdev(0x55934f276400 td/ceph-helpers/0/block) open stat got: (1) Operation not permitted 2026-02-25T22:31:40.656 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:40.647+0000 7fb9b8656a40 -1 bluestore(td/ceph-helpers/0) _read_fsid unparsable uuid 2026-02-25T22:31:41.421 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/ceph-helpers/0/keyring 2026-02-25T22:31:41.421 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-25T22:31:41.422 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-25T22:31:41.422 INFO:tasks.workunit.client.0.trial188.stdout:adding osd0 key to auth repository 2026-02-25T22:31:41.423 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/ceph-helpers/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-25T22:31:41.548 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-25T22:31:41.549 INFO:tasks.workunit.client.0.trial188.stdout:start osd.0 2026-02-25T22:31:41.549 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --cluster ceph --fsid=bd5a7b34-f18d-43a5-b117-7be57f98d09a --auth-supported=none --mon-host=127.0.0.1:7109 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/ceph-helpers/0 --osd-journal=td/ceph-helpers/0/journal --chdir= --run-dir=td/ceph-helpers '--admin-socket=/tmp/ceph-asok.55572/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/ceph-helpers/$name.log' '--pid-file=td/ceph-helpers/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-25T22:31:41.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-25T22:31:41.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-25T22:31:41.550 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-25T22:31:41.573 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:41.563+0000 7f8c84138a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:41.575 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:41.567+0000 7f8c84138a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:41.576 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:41.567+0000 7f8c84138a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-25T22:31:41.656 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-25T22:31:41.656 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-25T22:31:41.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-25T22:31:41.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-25T22:31:41.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-25T22:31:41.657 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:41.657 INFO:tasks.workunit.client.0.trial188.stdout:0 2026-02-25T22:31:41.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-25T22:31:41.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:41.658 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:41.773 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:42.107 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:42.099+0000 7f8c84138a40 -1 Falling back to public interface 2026-02-25T22:31:42.774 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:42.775 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:42.775 INFO:tasks.workunit.client.0.trial188.stdout:1 2026-02-25T22:31:42.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-25T22:31:42.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:42.776 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:42.883 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-25T22:31:43.087 INFO:tasks.workunit.client.0.trial188.stderr:2026-02-25T22:31:43.079+0000 7f8c84138a40 -1 osd.0 0 log_to_monitors true 2026-02-25T22:31:43.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-25T22:31:43.884 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-25T22:31:43.885 INFO:tasks.workunit.client.0.trial188.stdout:2 2026-02-25T22:31:43.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-25T22:31:43.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-25T22:31:43.885 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-25T22:31:44.100 INFO:tasks.workunit.client.0.trial188.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6800/703747668,v1:127.0.0.1:6801/703747668] [v2:127.0.0.1:6802/703747668,v1:127.0.0.1:6803/703747668] exists,up 1c2f7334-02c4-4bac-bff3-9cf34879eeb9 2026-02-25T22:31:44.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-25T22:31:44.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-25T22:31:44.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-25T22:31:44.101 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1947: test_pg_scrub: create_rbd_pool 2026-02-25T22:31:44.102 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:526: create_rbd_pool: ceph osd pool delete rbd rbd --yes-i-really-really-mean-it 2026-02-25T22:31:44.305 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' does not exist 2026-02-25T22:31:44.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:527: create_rbd_pool: create_pool rbd 4 2026-02-25T22:31:44.317 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:532: create_pool: ceph osd pool create rbd 4 2026-02-25T22:31:44.545 INFO:tasks.workunit.client.0.trial188.stderr:pool 'rbd' already exists 2026-02-25T22:31:44.554 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:533: create_pool: sleep 1 2026-02-25T22:31:45.556 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:528: create_rbd_pool: rbd pool init rbd 2026-02-25T22:31:45.859 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1948: test_pg_scrub: wait_for_clean 2026-02-25T22:31:45.860 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1647: wait_for_clean: local cmd= 2026-02-25T22:31:45.860 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1648: wait_for_clean: local num_active_clean=-1 2026-02-25T22:31:45.860 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1649: wait_for_clean: local cur_active_clean 2026-02-25T22:31:45.860 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: get_timeout_delays 90 .1 2026-02-25T22:31:45.860 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:45.860 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:45.860 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:45.861 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:45.861 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:45.954 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '4.5') 2026-02-25T22:31:45.954 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1650: wait_for_clean: local -a delays 2026-02-25T22:31:45.954 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1651: wait_for_clean: local -i loop=0 2026-02-25T22:31:45.954 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1653: wait_for_clean: flush_pg_stats 2026-02-25T22:31:45.954 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:31:45.954 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:46.163 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:31:46.163 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:46.163 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:46.163 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:46.243 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836484 2026-02-25T22:31:46.244 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836484 2026-02-25T22:31:46.244 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836484' 2026-02-25T22:31:46.244 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:46.244 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836484 2026-02-25T22:31:46.244 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:46.246 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:31:46.246 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836484 2026-02-25T22:31:46.246 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:46.247 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836484 2026-02-25T22:31:46.247 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836484 2026-02-25T22:31:46.248 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836484' 2026-02-25T22:31:46.248 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:46.458 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836482 -lt 21474836484 2026-02-25T22:31:46.458 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:47.459 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:31:47.460 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:47.682 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836484 -lt 21474836484 2026-02-25T22:31:47.682 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: get_num_pgs 2026-02-25T22:31:47.683 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:47.683 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:47.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1654: wait_for_clean: test 4 == 0 2026-02-25T22:31:47.970 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1658: wait_for_clean: true 2026-02-25T22:31:47.970 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: get_num_active_clean 2026-02-25T22:31:47.970 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1355: get_num_active_clean: local expression 2026-02-25T22:31:47.970 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1356: get_num_active_clean: expression+='select(contains("active") and contains("clean")) | ' 2026-02-25T22:31:47.970 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1357: get_num_active_clean: expression+='select(contains("stale") | not)' 2026-02-25T22:31:47.970 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1358: get_num_active_clean: ceph --format json pg dump pgs 2026-02-25T22:31:47.970 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1359: get_num_active_clean: jq '.pg_stats | [.[] | .state | select(contains("active") and contains("clean")) | select(contains("stale") | not)] | length' 2026-02-25T22:31:48.177 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1662: wait_for_clean: cur_active_clean=4 2026-02-25T22:31:48.177 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: get_num_pgs 2026-02-25T22:31:48.178 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: ceph --format json status 2026-02-25T22:31:48.178 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1416: get_num_pgs: jq .pgmap.num_pgs 2026-02-25T22:31:48.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: test 4 = 4 2026-02-25T22:31:48.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1663: wait_for_clean: break 2026-02-25T22:31:48.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1678: wait_for_clean: return 0 2026-02-25T22:31:48.460 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1949: test_pg_scrub: pg_scrub 1.0 2026-02-25T22:31:48.461 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-25T22:31:48.461 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-25T22:31:48.461 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-25T22:31:48.461 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 90 1 3 2026-02-25T22:31:48.461 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:48.461 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:48.462 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:48.462 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:48.462 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:48.732 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3' '3') 2026-02-25T22:31:48.733 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-25T22:31:48.733 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-25T22:31:48.733 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-25T22:31:48.733 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=300 2026-02-25T22:31:48.733 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:48.950 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:31:48.951 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:48.951 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:48.951 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 300 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:49.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq=21474836486 2026-02-25T22:31:49.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z 21474836486 2026-02-25T22:31:49.028 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2262: flush_pg_stats: seqs=' 0-21474836486' 2026-02-25T22:31:49.029 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2265: flush_pg_stats: for s in $seqs 2026-02-25T22:31:49.029 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: echo 0-21474836486 2026-02-25T22:31:49.029 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: cut -d - -f 1 2026-02-25T22:31:49.030 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2266: flush_pg_stats: osd=0 2026-02-25T22:31:49.031 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: echo 0-21474836486 2026-02-25T22:31:49.031 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: cut -d - -f 2 2026-02-25T22:31:49.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2267: flush_pg_stats: seq=21474836486 2026-02-25T22:31:49.032 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2268: flush_pg_stats: echo 'waiting osd.0 seq 21474836486' 2026-02-25T22:31:49.033 INFO:tasks.workunit.client.0.trial188.stdout:waiting osd.0 seq 21474836486 2026-02-25T22:31:49.033 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:49.256 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836484 -lt 21474836486 2026-02-25T22:31:49.256 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:50.258 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 300 -eq 0 ']' 2026-02-25T22:31:50.258 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:50.469 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836484 -lt 21474836486 2026-02-25T22:31:50.469 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2270: flush_pg_stats: sleep 1 2026-02-25T22:31:51.470 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2271: flush_pg_stats: '[' 299 -eq 0 ']' 2026-02-25T22:31:51.471 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: ceph osd last-stat-seq 0 2026-02-25T22:31:51.692 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2269: flush_pg_stats: test 21474836487 -lt 21474836486 2026-02-25T22:31:51.692 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-25T22:31:51.692 INFO:tasks.workunit.client.0.trial188.stdout:#---------- 1.0 loop 0 2026-02-25T22:31:51.693 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-25T22:31:51.693 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-25T22:31:51.693 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-25T22:31:51.693 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-25T22:31:51.693 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-25T22:31:51.693 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-25T22:31:51.779 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state=active+clean 2026-02-25T22:31:51.779 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ active+clean == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-25T22:31:51.779 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: break 2026-02-25T22:31:51.779 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1719: wait_for_pg_clean: return 0 2026-02-25T22:31:51.780 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: get_last_scrub_stamp 1.0 2026-02-25T22:31:51.780 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:31:51.780 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:31:51.780 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:31:51.780 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:31:51.984 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1926: pg_scrub: local last_scrub=2026-02-25T22:31:44.487876+0000 2026-02-25T22:31:51.985 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1927: pg_scrub: ceph pg scrub 1.0 2026-02-25T22:31:52.190 INFO:tasks.workunit.client.0.trial188.stderr:instructing pg 1.0 on osd.0 to scrub 2026-02-25T22:31:52.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1928: pg_scrub: wait_for_scrub 1.0 2026-02-25T22:31:44.487876+0000 2026-02-25T22:31:52.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2063: wait_for_scrub: local pgid=1.0 2026-02-25T22:31:52.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2064: wait_for_scrub: local last_scrub=2026-02-25T22:31:44.487876+0000 2026-02-25T22:31:52.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2065: wait_for_scrub: local sname=last_scrub_stamp 2026-02-25T22:31:52.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i=0 )) 2026-02-25T22:31:52.201 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:31:52.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:31:52.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:31:52.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:31:52.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:31:52.202 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:31:52.416 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-25T22:31:44.487876+0000 '>' 2026-02-25T22:31:44.487876+0000 2026-02-25T22:31:52.416 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-25T22:31:53.418 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-25T22:31:53.418 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:31:53.419 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:31:53.419 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:31:53.419 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:31:53.419 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:31:53.419 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:31:53.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-25T22:31:44.487876+0000 '>' 2026-02-25T22:31:44.487876+0000 2026-02-25T22:31:53.634 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2071: wait_for_scrub: sleep 1 2026-02-25T22:31:54.635 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i++ )) 2026-02-25T22:31:54.635 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2067: wait_for_scrub: (( i < 300 )) 2026-02-25T22:31:54.635 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: get_last_scrub_stamp 1.0 last_scrub_stamp 2026-02-25T22:31:54.636 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1517: get_last_scrub_stamp: local pgid=1.0 2026-02-25T22:31:54.636 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1518: get_last_scrub_stamp: local sname=last_scrub_stamp 2026-02-25T22:31:54.636 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1519: get_last_scrub_stamp: ceph --format json pg dump pgs 2026-02-25T22:31:54.636 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1520: get_last_scrub_stamp: jq -r '.pg_stats | .[] | select(.pgid=="1.0") | .last_scrub_stamp' 2026-02-25T22:31:54.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2068: wait_for_scrub: test 2026-02-25T22:31:53.038275+0000 '>' 2026-02-25T22:31:44.487876+0000 2026-02-25T22:31:54.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2069: wait_for_scrub: return 0 2026-02-25T22:31:54.853 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1950: test_pg_scrub: kill_daemons td/ceph-helpers KILL osd 2026-02-25T22:31:54.854 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-25T22:31:54.854 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-25T22:31:54.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-25T22:31:54.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-25T22:31:54.854 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-25T22:31:54.961 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-25T22:31:54.961 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: test_pg_scrub: WAIT_FOR_CLEAN_TIMEOUT=10 2026-02-25T22:31:54.961 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: test_pg_scrub: TIMEOUT=2 2026-02-25T22:31:54.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1951: test_pg_scrub: pg_scrub 1.0 2026-02-25T22:31:54.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1923: pg_scrub: local pgid=1.0 2026-02-25T22:31:54.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1925: pg_scrub: wait_for_pg_clean 1.0 2026-02-25T22:31:54.962 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1702: wait_for_pg_clean: local pg_id=1.0 2026-02-25T22:31:54.962 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: get_timeout_delays 10 1 3 2026-02-25T22:31:54.963 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-25T22:31:54.963 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-25T22:31:54.963 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-25T22:31:54.963 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-25T22:31:54.963 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-25T22:31:54.995 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: delays=('1' '2' '3' '3' '1') 2026-02-25T22:31:54.995 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1703: wait_for_pg_clean: local -a delays 2026-02-25T22:31:54.996 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1704: wait_for_pg_clean: local -i loop=0 2026-02-25T22:31:54.996 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1706: wait_for_pg_clean: flush_pg_stats 2026-02-25T22:31:54.996 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2253: flush_pg_stats: local timeout=2 2026-02-25T22:31:54.996 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ceph osd ls 2026-02-25T22:31:55.208 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2255: flush_pg_stats: ids=0 2026-02-25T22:31:55.208 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2256: flush_pg_stats: seqs= 2026-02-25T22:31:55.208 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2257: flush_pg_stats: for osd in $ids 2026-02-25T22:31:55.209 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: timeout 2 ceph tell osd.0 flush_pg_stats 2026-02-25T22:31:57.217 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: seq= 2026-02-25T22:31:57.217 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2258: flush_pg_stats: true 2026-02-25T22:31:57.217 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2259: flush_pg_stats: test -z '' 2026-02-25T22:31:57.217 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2260: flush_pg_stats: continue 2026-02-25T22:31:57.217 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-25T22:31:57.217 INFO:tasks.workunit.client.0.trial188.stdout:#---------- 1.0 loop 0 2026-02-25T22:31:57.218 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 0' 2026-02-25T22:31:57.218 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-25T22:31:57.218 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-25T22:31:57.218 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-25T22:31:57.218 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-25T22:31:57.218 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-26T01:28:23.127 DEBUG:teuthology.orchestra.run:got remote process result: 124 2026-02-26T01:28:23.135 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: pg_state= 2026-02-26T01:28:23.135 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1: is_pg_clean: teardown td/ceph-helpers 1 2026-02-26T01:28:23.135 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/ceph-helpers 2026-02-26T01:28:23.135 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=1 2026-02-26T01:28:23.135 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/ceph-helpers KILL 2026-02-26T01:28:23.135 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-26T01:28:23.136 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-26T01:28:23.136 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-26T01:28:23.136 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-26T01:28:23.136 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-26T01:28:23.223 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-26T01:28:23.223 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-26T01:28:23.225 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-26T01:28:23.225 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-26T01:28:23.226 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-26T01:28:23.226 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-26T01:28:23.227 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-26T01:28:23.228 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T01:28:23.228 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-26T01:28:23.229 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-26T01:28:23.229 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-26T01:28:23.230 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-26T01:28:23.232 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 1 = 1 ']' 2026-02-26T01:28:23.232 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:181: teardown: '[' -n '' ']' 2026-02-26T01:28:23.232 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:185: teardown: mkdir -p /home/ubuntu/cephtest/archive/log 2026-02-26T01:28:23.233 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:186: teardown: mv td/ceph-helpers/mgr.x.log td/ceph-helpers/mon.a.log td/ceph-helpers/osd.0.log /home/ubuntu/cephtest/archive/log 2026-02-26T01:28:23.234 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/ceph-helpers 2026-02-26T01:28:23.239 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-26T01:28:23.239 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-26T01:28:23.239 INFO:tasks.workunit.client.0.trial188.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.55572 2026-02-26T01:28:23.239 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.55572 2026-02-26T01:28:23.240 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-26T01:28:23.240 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-26T01:28:23.241 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1575: is_pg_clean: [[ '' == \a\c\t\i\v\e\+\c\l\e\a\n* ]] 2026-02-26T01:28:23.241 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1711: wait_for_pg_clean: (( 0 >= 5 )) 2026-02-26T01:28:23.241 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1716: wait_for_pg_clean: sleep 1 2026-02-26T01:28:24.241 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1717: wait_for_pg_clean: loop+=1 2026-02-26T01:28:24.242 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1708: wait_for_pg_clean: true 2026-02-26T01:28:24.242 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1709: wait_for_pg_clean: echo '#---------- 1.0 loop 1' 2026-02-26T01:28:24.242 INFO:tasks.workunit.client.0.trial188.stdout:#---------- 1.0 loop 1 2026-02-26T01:28:24.243 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1710: wait_for_pg_clean: is_pg_clean 1.0 2026-02-26T01:28:24.243 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1572: is_pg_clean: local pgid=1.0 2026-02-26T01:28:24.243 INFO:tasks.workunit.client.0.trial188.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1573: is_pg_clean: local pg_state 2026-02-26T01:28:24.243 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: ceph pg 1.0 query 2026-02-26T01:28:24.243 INFO:tasks.workunit.client.0.trial188.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1574: is_pg_clean: jq -r '.state ' 2026-02-26T01:29:23.135 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2026-02-26T01:30:23.136 DEBUG:teuthology.orchestra.run:timed out waiting; will kill: 2026-02-26T01:30:23.137 INFO:tasks.workunit:Stopping ['misc'] on client.0... 2026-02-26T01:30:23.137 DEBUG:teuthology.orchestra.run.trial188:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-26T01:30:23.458 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) ^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_Ericmzhang_ceph_61d4431d3ec879bd98659388b3655acf3d392418/qa/tasks/workunit.py", line 144, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/github.com_Ericmzhang_ceph_61d4431d3ec879bd98659388b3655acf3d392418/qa/tasks/workunit.py", line 292, in _spawn_on_all_clients with parallel() as p: File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/github.com_Ericmzhang_ceph_61d4431d3ec879bd98659388b3655acf3d392418/qa/tasks/workunit.py", line 433, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/remote.py", line 596, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 461, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test misc/test-ceph-helpers.sh) on trial188 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61d4431d3ec879bd98659388b3655acf3d392418 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh' 2026-02-26T01:30:23.464 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-26T01:30:23.481 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-26T01:30:23.482 DEBUG:teuthology.orchestra.run.trial188:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-26T01:30:23.591 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-26T01:30:23.603 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-26T01:30:23.603 DEBUG:teuthology.orchestra.run.trial188:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-26T01:30:23.649 INFO:teuthology.orchestra.run.trial188.stdout: remote refid st t when poll reach delay offset jitter 2026-02-26T01:30:23.649 INFO:teuthology.orchestra.run.trial188.stdout:============================================================================== 2026-02-26T01:30:23.649 INFO:teuthology.orchestra.run.trial188.stdout:*ool-44c5de70.dy .PPS. 1 u 98 256 377 5.349 +1.742 1.961 2026-02-26T01:30:23.649 INFO:teuthology.orchestra.run.trial188.stdout:+108.181.220.94 225.254.30.190 4 u 252 256 377 42.435 +2.745 3.701 2026-02-26T01:30:23.649 INFO:teuthology.orchestra.run.trial188.stdout:+nyc2.us.ntp.li 17.253.2.35 2 u 215 256 377 3.247 +1.320 1.423 2026-02-26T01:30:23.649 INFO:teuthology.orchestra.run.trial188.stdout:+144.202.66.214 198.46.254.130 3 u 26 256 377 44.380 +2.749 1.977 2026-02-26T01:30:23.650 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-26T01:30:23.662 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-26T01:30:23.662 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-26T01:30:23.676 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-26T01:30:23.691 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-26T01:30:23.707 INFO:teuthology.task.internal:Duration was 11547.169406 seconds 2026-02-26T01:30:23.708 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-26T01:30:23.723 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-26T01:30:23.723 DEBUG:teuthology.orchestra.run.trial188:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-26T01:30:23.750 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-26T01:30:23.750 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial188.front.sepia.ceph.com 2026-02-26T01:30:23.750 DEBUG:teuthology.orchestra.run.trial188:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-26T01:30:23.798 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-26T01:30:23.798 DEBUG:teuthology.orchestra.run.trial188:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T01:30:24.064 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-26T01:30:24.065 DEBUG:teuthology.orchestra.run.trial188:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-26T01:30:24.073 INFO:teuthology.orchestra.run.trial188.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-26T01:30:24.074 INFO:teuthology.orchestra.run.trial188.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-26T01:30:24.074 INFO:teuthology.orchestra.run.trial188.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-26T01:30:24.075 INFO:teuthology.orchestra.run.trial188.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-26T01:30:24.075 INFO:teuthology.orchestra.run.trial188.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-26T01:30:24.146 INFO:teuthology.orchestra.run.trial188.stderr: 81.8% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-26T01:30:24.148 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-26T01:30:24.163 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-26T01:30:24.163 DEBUG:teuthology.orchestra.run.trial188:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-26T01:30:24.196 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-26T01:30:24.212 DEBUG:teuthology.orchestra.run.trial188:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-26T01:30:24.248 INFO:teuthology.orchestra.run.trial188.stdout:kernel.core_pattern = core 2026-02-26T01:30:24.257 DEBUG:teuthology.orchestra.run.trial188:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-26T01:30:24.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T01:30:24.303 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-26T01:30:24.319 INFO:teuthology.task.internal:Transferring archived files... 2026-02-26T01:30:24.323 DEBUG:teuthology.misc:Transferring archived files from trial188:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/emzhang-2026-02-25_22:10:53-rados:standalone-wip-mon-colocate-distro-default-trial/70701/remote/trial188 2026-02-26T01:30:24.324 DEBUG:teuthology.orchestra.run.trial188:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-26T01:30:37.795 INFO:teuthology.task.internal:Removing archive directory... 2026-02-26T01:30:37.795 DEBUG:teuthology.orchestra.run.trial188:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-26T01:30:37.888 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-26T01:30:37.901 INFO:teuthology.task.internal:Not uploading archives. 2026-02-26T01:30:37.901 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-26T01:30:37.914 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-26T01:30:37.915 DEBUG:teuthology.orchestra.run.trial188:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-26T01:30:37.919 INFO:teuthology.orchestra.run.trial188.stdout: 1311264 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Feb 26 01:30 /home/ubuntu/cephtest 2026-02-26T01:30:37.919 INFO:teuthology.orchestra.run.trial188.stdout: 1321456 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 25 22:21 /home/ubuntu/cephtest/mnt.0 2026-02-26T01:30:37.919 INFO:teuthology.orchestra.run.trial188.stdout: 1321457 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 25 22:28 /home/ubuntu/cephtest/mnt.0/client.0 2026-02-26T01:30:37.919 INFO:teuthology.orchestra.run.trial188.stdout: 1333531 4 drwxrwxr-x 3 ubuntu ubuntu 4096 Feb 25 22:28 /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-26T01:30:37.920 INFO:teuthology.orchestra.run.trial188.stdout: 1333534 4 drwxrwxr-x 2 ubuntu ubuntu 4096 Feb 26 01:28 /home/ubuntu/cephtest/mnt.0/client.0/tmp/td 2026-02-26T01:30:37.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-26T01:30:37.920 INFO:teuthology.orchestra.run.trial188.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2026-02-26T01:30:37.921 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/run_tasks.py", line 160, in run_tasks suppress = manager.__exit__(*exc_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/contextlib.py", line 144, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/task/internal/__init__.py", line 53, in base run.wait( File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 485, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on trial188 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2026-02-26T01:30:37.922 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-26T01:30:37.934 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-26T01:30:37.948 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: CommandFailedError: Command failed on trial188 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest' 2026-02-26T01:30:37.952 INFO:teuthology.run:Summary data: description: rados:standalone/{supported-random-distro$/{ubuntu_latest} workloads/misc} duration: 11547.169406414032 failure_reason: 'Command failed (workunit test misc/test-ceph-helpers.sh) on trial188 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=61d4431d3ec879bd98659388b3655acf3d392418 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/misc/test-ceph-helpers.sh''' flavor: default owner: scheduled_emzhang@soko04.front.sepia.ceph.com sentry_event: null status: fail success: false 2026-02-26T01:30:37.952 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-26T01:30:39.411 INFO:teuthology.run:FAIL