2026-02-18T00:38:39.119 INFO:root:teuthology version: 1.2.4.dev9+gfa17720d0 2026-02-18T00:38:39.135 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-18T00:38:39.216 INFO:teuthology.run:Config: archive_path: /home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:14:21-rados-20.2.1_2.17.26-baseline_2-distro-default-trial/54187 branch: 20.2.1_2.17.26-baseline_2 description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon-stretch} email: yweinste@redhat.com first_in_suite: false flavor: default job_id: '54187' kernel: branch: distro kdb: true sha1: distro ktype: distro last_in_suite: false machine_type: trial name: yuriw-2026-02-17_21:14:21-rados-20.2.1_2.17.26-baseline_2-distro-default-trial no_nested_subset: false openstack: - volumes: count: 3 size: 10 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: 20.2.1_2.17.26-baseline_2 ceph: conf: mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: adc1923395f2bcd6ae89b5d331d278364d1848ed ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: adc1923395f2bcd6ae89b5d331d278364d1848ed workunit: branch: 20.2.1_2.17.26-baseline_2 sha1: adc1923395f2bcd6ae89b5d331d278364d1848ed owner: scheduled_yuriw@soko04.front.sepia.ceph.com priority: 69 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.x - osd.0 - osd.1 - osd.2 - client.0 seed: 1775 sha1: adc1923395f2bcd6ae89b5d331d278364d1848ed sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: 20.2.1_2.17.26-baseline_2 suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_adc1923395f2bcd6ae89b5d331d278364d1848ed/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: adc1923395f2bcd6ae89b5d331d278364d1848ed targets: trial038.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCwHe7IL7IfWfIqP45l5q886pYn/RtEatA7wjsMfBj9xbhZW1bJHJzu42YBPxsQSivCunzgmybRmcA3baT/fRqA= tasks: - install: null - workunit: basedir: qa/standalone clients: all: - mon-stretch teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: fa17720d0088c3ac28e473468bfc79eeaff5cd38 timestamp: 2026-02-17_21:14:21 tube: trial user: yuriw verbose: true worker_log: /home/teuthworker/mnt/teuthology/worker_logs/dispatcher.trial.3682 2026-02-18T00:38:39.216 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_adc1923395f2bcd6ae89b5d331d278364d1848ed/qa; will attempt to use it 2026-02-18T00:38:39.217 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_adc1923395f2bcd6ae89b5d331d278364d1848ed/qa/tasks 2026-02-18T00:38:39.217 INFO:teuthology.run_tasks:Running task internal.check_packages... 2026-02-18T00:38:39.218 INFO:teuthology.task.internal:Checking packages... 2026-02-18T00:38:39.218 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'adc1923395f2bcd6ae89b5d331d278364d1848ed' 2026-02-18T00:38:39.218 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2026-02-18T00:38:39.218 INFO:teuthology.packaging:ref: None 2026-02-18T00:38:39.218 INFO:teuthology.packaging:tag: None 2026-02-18T00:38:39.218 INFO:teuthology.packaging:branch: 20.2.1_2.17.26-baseline_2 2026-02-18T00:38:39.218 INFO:teuthology.packaging:sha1: adc1923395f2bcd6ae89b5d331d278364d1848ed 2026-02-18T00:38:39.218 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=20.2.1_2.17.26-baseline_2 2026-02-18T00:38:39.345 INFO:teuthology.task.internal:Found packages for ceph version 20.2.0-670-gadc19233-1jammy 2026-02-18T00:38:39.346 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2026-02-18T00:38:39.359 INFO:teuthology.task.internal:no buildpackages task found 2026-02-18T00:38:39.359 INFO:teuthology.run_tasks:Running task internal.save_config... 2026-02-18T00:38:39.373 INFO:teuthology.task.internal:Saving configuration 2026-02-18T00:38:39.381 INFO:teuthology.run_tasks:Running task internal.check_lock... 2026-02-18T00:38:39.393 INFO:teuthology.task.internal.check_lock:Checking locks... 2026-02-18T00:38:39.473 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'trial038.front.sepia.ceph.com', 'description': '/home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:14:21-rados-20.2.1_2.17.26-baseline_2-distro-default-trial/54187', 'up': True, 'machine_type': 'trial', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2026-02-18 00:35:16.059714', 'locked_by': 'scheduled_yuriw@soko04.front.sepia.ceph.com', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBzTKhg2y1zTVTSAhgsKMzFkZe48H43VTwLlJuoF06Mi'} 2026-02-18T00:38:39.473 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2026-02-18T00:38:39.485 INFO:teuthology.task.internal:roles: ubuntu@trial038.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'osd.0', 'osd.1', 'osd.2', 'client.0'] 2026-02-18T00:38:39.485 INFO:teuthology.run_tasks:Running task console_log... 2026-02-18T00:38:41.047 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7920737e27a0>, signals=[15]) 2026-02-18T00:38:41.047 INFO:teuthology.run_tasks:Running task internal.connect... 2026-02-18T00:38:41.059 INFO:teuthology.task.internal:Opening connections... 2026-02-18T00:38:41.060 DEBUG:teuthology.task.internal:connecting to ubuntu@trial038.front.sepia.ceph.com 2026-02-18T00:38:41.061 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-18T00:38:41.149 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2026-02-18T00:38:41.161 DEBUG:teuthology.orchestra.run.trial038:> uname -m 2026-02-18T00:38:41.229 INFO:teuthology.orchestra.run.trial038.stdout:x86_64 2026-02-18T00:38:41.229 DEBUG:teuthology.orchestra.run.trial038:> cat /etc/os-release 2026-02-18T00:38:41.273 INFO:teuthology.orchestra.run.trial038.stdout:PRETTY_NAME="Ubuntu 22.04.5 LTS" 2026-02-18T00:38:41.273 INFO:teuthology.orchestra.run.trial038.stdout:NAME="Ubuntu" 2026-02-18T00:38:41.273 INFO:teuthology.orchestra.run.trial038.stdout:VERSION_ID="22.04" 2026-02-18T00:38:41.273 INFO:teuthology.orchestra.run.trial038.stdout:VERSION="22.04.5 LTS (Jammy Jellyfish)" 2026-02-18T00:38:41.273 INFO:teuthology.orchestra.run.trial038.stdout:VERSION_CODENAME=jammy 2026-02-18T00:38:41.273 INFO:teuthology.orchestra.run.trial038.stdout:ID=ubuntu 2026-02-18T00:38:41.274 INFO:teuthology.orchestra.run.trial038.stdout:ID_LIKE=debian 2026-02-18T00:38:41.274 INFO:teuthology.orchestra.run.trial038.stdout:HOME_URL="https://www.ubuntu.com/" 2026-02-18T00:38:41.274 INFO:teuthology.orchestra.run.trial038.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2026-02-18T00:38:41.274 INFO:teuthology.orchestra.run.trial038.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2026-02-18T00:38:41.274 INFO:teuthology.orchestra.run.trial038.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2026-02-18T00:38:41.274 INFO:teuthology.orchestra.run.trial038.stdout:UBUNTU_CODENAME=jammy 2026-02-18T00:38:41.275 INFO:teuthology.lock.ops:Updating trial038.front.sepia.ceph.com on lock server 2026-02-18T00:38:41.367 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2026-02-18T00:38:41.380 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2026-02-18T00:38:41.392 INFO:teuthology.task.internal:Checking for old test directory... 2026-02-18T00:38:41.392 DEBUG:teuthology.orchestra.run.trial038:> test '!' -e /home/ubuntu/cephtest 2026-02-18T00:38:41.395 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2026-02-18T00:38:41.404 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2026-02-18T00:38:41.404 DEBUG:teuthology.orchestra.run.trial038:> test -z $(ls -A /var/lib/ceph) 2026-02-18T00:38:41.441 INFO:teuthology.orchestra.run.trial038.stderr:ls: cannot access '/var/lib/ceph': No such file or directory 2026-02-18T00:38:41.442 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2026-02-18T00:38:41.539 INFO:teuthology.run_tasks:Running task kernel... 2026-02-18T00:38:41.558 INFO:teuthology.task.kernel:normalize config orig: {'branch': 'distro', 'kdb': True, 'sha1': 'distro'} 2026-02-18T00:38:41.558 INFO:teuthology.task.kernel:config {'mon.a': {'branch': 'distro', 'kdb': True, 'sha1': 'distro'}}, timeout 300 2026-02-18T00:38:41.558 DEBUG:teuthology.orchestra.run.trial038:> test -f /run/.containerenv -o -f /.dockerenv 2026-02-18T00:38:41.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-18T00:38:41.561 DEBUG:teuthology.orchestra.run.trial038:> uname -r 2026-02-18T00:38:41.604 INFO:teuthology.orchestra.run.trial038.stdout:5.15.0-164-generic 2026-02-18T00:38:41.605 INFO:teuthology.task.kernel:Running kernel on trial038: 5.15.0-164-generic 2026-02-18T00:38:41.605 DEBUG:teuthology.orchestra.run.trial038:> sudo apt-get clean 2026-02-18T00:38:41.666 DEBUG:teuthology.orchestra.run.trial038:> sudo apt-get update 2026-02-18T00:38:41.863 INFO:teuthology.orchestra.run.trial038.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-18T00:38:41.870 INFO:teuthology.orchestra.run.trial038.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2026-02-18T00:38:41.909 INFO:teuthology.orchestra.run.trial038.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2026-02-18T00:38:41.925 INFO:teuthology.orchestra.run.trial038.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2026-02-18T00:38:42.012 INFO:teuthology.orchestra.run.trial038.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [3240 kB] 2026-02-18T00:38:42.079 INFO:teuthology.orchestra.run.trial038.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [494 kB] 2026-02-18T00:38:42.082 INFO:teuthology.orchestra.run.trial038.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [19.2 kB] 2026-02-18T00:38:42.083 INFO:teuthology.orchestra.run.trial038.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1256 kB] 2026-02-18T00:38:42.090 INFO:teuthology.orchestra.run.trial038.stdout:Get:9 https://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [314 kB] 2026-02-18T00:38:42.092 INFO:teuthology.orchestra.run.trial038.stdout:Get:10 https://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [5295 kB] 2026-02-18T00:38:42.117 INFO:teuthology.orchestra.run.trial038.stdout:Get:11 https://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [1001 kB] 2026-02-18T00:38:42.120 INFO:teuthology.orchestra.run.trial038.stdout:Get:12 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 Packages [2975 kB] 2026-02-18T00:38:42.129 INFO:teuthology.orchestra.run.trial038.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [14.1 kB] 2026-02-18T00:38:42.130 INFO:teuthology.orchestra.run.trial038.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [1017 kB] 2026-02-18T00:38:45.481 INFO:teuthology.orchestra.run.trial038.stdout:Fetched 16.0 MB in 1s (12.9 MB/s) 2026-02-18T00:38:46.000 INFO:teuthology.orchestra.run.trial038.stdout:Reading package lists... 2026-02-18T00:38:46.014 DEBUG:teuthology.orchestra.run.trial038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-18T00:38:46.047 INFO:teuthology.orchestra.run.trial038.stdout:Reading package lists... 2026-02-18T00:38:46.155 INFO:teuthology.orchestra.run.trial038.stdout:Building dependency tree... 2026-02-18T00:38:46.156 INFO:teuthology.orchestra.run.trial038.stdout:Reading state information... 2026-02-18T00:38:46.241 INFO:teuthology.orchestra.run.trial038.stdout:The following packages were automatically installed and are no longer required: 2026-02-18T00:38:46.241 INFO:teuthology.orchestra.run.trial038.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-18T00:38:46.241 INFO:teuthology.orchestra.run.trial038.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-18T00:38:46.242 INFO:teuthology.orchestra.run.trial038.stdout: sg3-utils-udev 2026-02-18T00:38:46.242 INFO:teuthology.orchestra.run.trial038.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-18T00:38:46.242 INFO:teuthology.orchestra.run.trial038.stdout:The following additional packages will be installed: 2026-02-18T00:38:46.242 INFO:teuthology.orchestra.run.trial038.stdout: linux-generic linux-headers-5.15.0-170 linux-headers-5.15.0-170-generic 2026-02-18T00:38:46.243 INFO:teuthology.orchestra.run.trial038.stdout: linux-headers-generic linux-image-5.15.0-170-generic 2026-02-18T00:38:46.243 INFO:teuthology.orchestra.run.trial038.stdout: linux-modules-5.15.0-170-generic linux-modules-extra-5.15.0-170-generic 2026-02-18T00:38:46.243 INFO:teuthology.orchestra.run.trial038.stdout:Suggested packages: 2026-02-18T00:38:46.243 INFO:teuthology.orchestra.run.trial038.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2026-02-18T00:38:46.256 INFO:teuthology.orchestra.run.trial038.stdout:The following NEW packages will be installed: 2026-02-18T00:38:46.256 INFO:teuthology.orchestra.run.trial038.stdout: linux-headers-5.15.0-170 linux-headers-5.15.0-170-generic 2026-02-18T00:38:46.256 INFO:teuthology.orchestra.run.trial038.stdout: linux-image-5.15.0-170-generic linux-modules-5.15.0-170-generic 2026-02-18T00:38:46.256 INFO:teuthology.orchestra.run.trial038.stdout: linux-modules-extra-5.15.0-170-generic 2026-02-18T00:38:46.257 INFO:teuthology.orchestra.run.trial038.stdout:The following packages will be upgraded: 2026-02-18T00:38:46.257 INFO:teuthology.orchestra.run.trial038.stdout: linux-generic linux-headers-generic linux-image-generic 2026-02-18T00:38:46.351 INFO:teuthology.orchestra.run.trial038.stdout:3 upgraded, 5 newly installed, 0 to remove and 53 not upgraded. 2026-02-18T00:38:46.351 INFO:teuthology.orchestra.run.trial038.stdout:Need to get 113 MB of archives. 2026-02-18T00:38:46.351 INFO:teuthology.orchestra.run.trial038.stdout:After this operation, 585 MB of additional disk space will be used. 2026-02-18T00:38:46.351 INFO:teuthology.orchestra.run.trial038.stdout:Get:1 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-170-generic amd64 5.15.0-170.180 [22.7 MB] 2026-02-18T00:38:46.520 INFO:teuthology.orchestra.run.trial038.stdout:Get:2 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-170-generic amd64 5.15.0-170.180 [11.6 MB] 2026-02-18T00:38:46.563 INFO:teuthology.orchestra.run.trial038.stdout:Get:3 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-170-generic amd64 5.15.0-170.180 [64.0 MB] 2026-02-18T00:38:46.807 INFO:teuthology.orchestra.run.trial038.stdout:Get:4 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.170.159 [1654 B] 2026-02-18T00:38:46.808 INFO:teuthology.orchestra.run.trial038.stdout:Get:5 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.170.159 [2474 B] 2026-02-18T00:38:46.808 INFO:teuthology.orchestra.run.trial038.stdout:Get:6 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-170 all 5.15.0-170.180 [12.4 MB] 2026-02-18T00:38:46.866 INFO:teuthology.orchestra.run.trial038.stdout:Get:7 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-170-generic amd64 5.15.0-170.180 [2828 kB] 2026-02-18T00:38:46.877 INFO:teuthology.orchestra.run.trial038.stdout:Get:8 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.170.159 [2316 B] 2026-02-18T00:38:47.075 INFO:teuthology.orchestra.run.trial038.stdout:Fetched 113 MB in 1s (183 MB/s) 2026-02-18T00:38:47.095 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package linux-modules-5.15.0-170-generic. 2026-02-18T00:38:47.263 INFO:teuthology.orchestra.run.trial038.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 131954 files and directories currently installed.) 2026-02-18T00:38:47.264 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../0-linux-modules-5.15.0-170-generic_5.15.0-170.180_amd64.deb ... 2026-02-18T00:38:47.265 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking linux-modules-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-18T00:38:47.618 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package linux-image-5.15.0-170-generic. 2026-02-18T00:38:47.625 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../1-linux-image-5.15.0-170-generic_5.15.0-170.180_amd64.deb ... 2026-02-18T00:38:47.643 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking linux-image-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-18T00:38:47.685 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-170-generic. 2026-02-18T00:38:47.691 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-170-generic_5.15.0-170.180_amd64.deb ... 2026-02-18T00:38:47.692 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking linux-modules-extra-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-18T00:38:48.805 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../3-linux-generic_5.15.0.170.159_amd64.deb ... 2026-02-18T00:38:48.805 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking linux-generic (5.15.0.170.159) over (5.15.0.164.159) ... 2026-02-18T00:38:48.827 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.170.159_amd64.deb ... 2026-02-18T00:38:48.827 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking linux-image-generic (5.15.0.170.159) over (5.15.0.164.159) ... 2026-02-18T00:38:48.846 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package linux-headers-5.15.0-170. 2026-02-18T00:38:48.853 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../5-linux-headers-5.15.0-170_5.15.0-170.180_all.deb ... 2026-02-18T00:38:48.853 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking linux-headers-5.15.0-170 (5.15.0-170.180) ... 2026-02-18T00:38:50.230 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package linux-headers-5.15.0-170-generic. 2026-02-18T00:38:50.242 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../6-linux-headers-5.15.0-170-generic_5.15.0-170.180_amd64.deb ... 2026-02-18T00:38:50.242 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking linux-headers-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-18T00:38:50.835 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.170.159_amd64.deb ... 2026-02-18T00:38:50.835 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking linux-headers-generic (5.15.0.170.159) over (5.15.0.164.159) ... 2026-02-18T00:38:50.862 INFO:teuthology.orchestra.run.trial038.stdout:Setting up linux-headers-5.15.0-170 (5.15.0-170.180) ... 2026-02-18T00:38:50.863 INFO:teuthology.orchestra.run.trial038.stdout:Setting up linux-headers-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-18T00:38:50.865 INFO:teuthology.orchestra.run.trial038.stdout:Setting up linux-headers-generic (5.15.0.170.159) ... 2026-02-18T00:38:50.866 INFO:teuthology.orchestra.run.trial038.stdout:Setting up linux-modules-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-18T00:38:51.641 INFO:teuthology.orchestra.run.trial038.stdout:Setting up linux-image-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-18T00:38:52.478 INFO:teuthology.orchestra.run.trial038.stdout:I: /boot/vmlinuz is now a symlink to vmlinuz-5.15.0-170-generic 2026-02-18T00:38:52.479 INFO:teuthology.orchestra.run.trial038.stdout:I: /boot/initrd.img is now a symlink to initrd.img-5.15.0-170-generic 2026-02-18T00:38:52.484 INFO:teuthology.orchestra.run.trial038.stdout:Setting up linux-modules-extra-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-18T00:38:53.286 INFO:teuthology.orchestra.run.trial038.stdout:Setting up linux-image-generic (5.15.0.170.159) ... 2026-02-18T00:38:53.287 INFO:teuthology.orchestra.run.trial038.stdout:Setting up linux-generic (5.15.0.170.159) ... 2026-02-18T00:38:53.288 INFO:teuthology.orchestra.run.trial038.stdout:Processing triggers for linux-image-5.15.0-170-generic (5.15.0-170.180) ... 2026-02-18T00:38:53.294 INFO:teuthology.orchestra.run.trial038.stdout:/etc/kernel/postinst.d/initramfs-tools: 2026-02-18T00:38:53.294 INFO:teuthology.orchestra.run.trial038.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-170-generic 2026-02-18T00:39:00.368 INFO:teuthology.orchestra.run.trial038.stdout:W: Possible missing firmware /lib/firmware/ast_dp501_fw.bin for module ast 2026-02-18T00:39:06.942 INFO:teuthology.orchestra.run.trial038.stdout:/etc/kernel/postinst.d/zz-update-grub: 2026-02-18T00:39:06.942 INFO:teuthology.orchestra.run.trial038.stdout:Sourcing file `/etc/default/grub' 2026-02-18T00:39:06.962 INFO:teuthology.orchestra.run.trial038.stdout:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-18T00:39:06.963 INFO:teuthology.orchestra.run.trial038.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-18T00:39:06.966 INFO:teuthology.orchestra.run.trial038.stdout:Generating grub configuration file ... 2026-02-18T00:39:07.047 INFO:teuthology.orchestra.run.trial038.stdout:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-18T00:39:07.054 INFO:teuthology.orchestra.run.trial038.stdout:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-18T00:39:07.227 INFO:teuthology.orchestra.run.trial038.stdout:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-18T00:39:07.232 INFO:teuthology.orchestra.run.trial038.stdout:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-18T00:39:07.305 INFO:teuthology.orchestra.run.trial038.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-18T00:39:07.305 INFO:teuthology.orchestra.run.trial038.stdout:Systems on them will not be added to the GRUB boot configuration. 2026-02-18T00:39:07.305 INFO:teuthology.orchestra.run.trial038.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-18T00:39:07.307 INFO:teuthology.orchestra.run.trial038.stdout:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-18T00:39:07.316 INFO:teuthology.orchestra.run.trial038.stdout:done 2026-02-18T00:39:07.739 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:39:07.739 INFO:teuthology.orchestra.run.trial038.stdout:Pending kernel upgrade! 2026-02-18T00:39:07.739 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:39:07.739 INFO:teuthology.orchestra.run.trial038.stdout:Running kernel version: 2026-02-18T00:39:07.739 INFO:teuthology.orchestra.run.trial038.stdout: 5.15.0-164-generic 2026-02-18T00:39:07.739 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:39:07.739 INFO:teuthology.orchestra.run.trial038.stdout:Diagnostics: 2026-02-18T00:39:07.739 INFO:teuthology.orchestra.run.trial038.stdout: The currently running kernel version is not the expected kernel version 5.15.0-170-generic. 2026-02-18T00:39:07.740 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:39:07.740 INFO:teuthology.orchestra.run.trial038.stdout:Restarting the system to load the new kernel will not be handled automatically, so you should consider rebooting. [Return] 2026-02-18T00:39:07.740 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:39:07.740 INFO:teuthology.orchestra.run.trial038.stdout:The processor microcode seems to be up-to-date. 2026-02-18T00:39:07.740 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:39:07.740 INFO:teuthology.orchestra.run.trial038.stdout:No services need to be restarted. 2026-02-18T00:39:07.740 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:39:07.740 INFO:teuthology.orchestra.run.trial038.stdout:No containers need to be restarted. 2026-02-18T00:39:07.740 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:39:07.740 INFO:teuthology.orchestra.run.trial038.stdout:No user sessions are running outdated binaries. 2026-02-18T00:39:07.740 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:39:07.740 INFO:teuthology.orchestra.run.trial038.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-18T00:39:08.404 DEBUG:teuthology.orchestra.run.trial038:> dpkg -s linux-image-generic 2026-02-18T00:39:08.455 INFO:teuthology.orchestra.run.trial038.stdout:Package: linux-image-generic 2026-02-18T00:39:08.456 INFO:teuthology.orchestra.run.trial038.stdout:Status: install ok installed 2026-02-18T00:39:08.456 INFO:teuthology.orchestra.run.trial038.stdout:Priority: optional 2026-02-18T00:39:08.456 INFO:teuthology.orchestra.run.trial038.stdout:Section: kernel 2026-02-18T00:39:08.456 INFO:teuthology.orchestra.run.trial038.stdout:Installed-Size: 21 2026-02-18T00:39:08.456 INFO:teuthology.orchestra.run.trial038.stdout:Maintainer: Ubuntu Kernel Team 2026-02-18T00:39:08.456 INFO:teuthology.orchestra.run.trial038.stdout:Architecture: amd64 2026-02-18T00:39:08.456 INFO:teuthology.orchestra.run.trial038.stdout:Source: linux-meta 2026-02-18T00:39:08.456 INFO:teuthology.orchestra.run.trial038.stdout:Version: 5.15.0.170.159 2026-02-18T00:39:08.456 INFO:teuthology.orchestra.run.trial038.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-18T00:39:08.457 INFO:teuthology.orchestra.run.trial038.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-18T00:39:08.457 INFO:teuthology.orchestra.run.trial038.stdout:Recommends: thermald 2026-02-18T00:39:08.457 INFO:teuthology.orchestra.run.trial038.stdout:Description: Generic Linux kernel image 2026-02-18T00:39:08.457 INFO:teuthology.orchestra.run.trial038.stdout: This package will always depend on the latest generic kernel image 2026-02-18T00:39:08.457 INFO:teuthology.orchestra.run.trial038.stdout: available. 2026-02-18T00:39:08.457 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-164-generic Expected: 5.15.0-170-generic 2026-02-18T00:39:08.457 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2026-02-18T00:39:08.458 DEBUG:teuthology.task.kernel:src is distro, skipping download 2026-02-18T00:39:08.458 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2026-02-18T00:39:08.458 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@trial038.front.sepia.ceph.com, path=None, version=distro) 2026-02-18T00:39:08.458 DEBUG:teuthology.orchestra.run.trial038:> sudo apt-get clean 2026-02-18T00:39:08.537 DEBUG:teuthology.orchestra.run.trial038:> sudo apt-get update 2026-02-18T00:39:08.725 INFO:teuthology.orchestra.run.trial038.stdout:Hit:1 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-18T00:39:08.729 INFO:teuthology.orchestra.run.trial038.stdout:Hit:2 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-18T00:39:08.740 INFO:teuthology.orchestra.run.trial038.stdout:Hit:3 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-18T00:39:08.750 INFO:teuthology.orchestra.run.trial038.stdout:Hit:4 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-18T00:39:09.556 INFO:teuthology.orchestra.run.trial038.stdout:Reading package lists... 2026-02-18T00:39:09.570 DEBUG:teuthology.orchestra.run.trial038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2026-02-18T00:39:09.603 INFO:teuthology.orchestra.run.trial038.stdout:Reading package lists... 2026-02-18T00:39:09.712 INFO:teuthology.orchestra.run.trial038.stdout:Building dependency tree... 2026-02-18T00:39:09.713 INFO:teuthology.orchestra.run.trial038.stdout:Reading state information... 2026-02-18T00:39:09.795 INFO:teuthology.orchestra.run.trial038.stdout:linux-image-generic is already the newest version (5.15.0.170.159). 2026-02-18T00:39:09.795 INFO:teuthology.orchestra.run.trial038.stdout:linux-image-generic set to manually installed. 2026-02-18T00:39:09.796 INFO:teuthology.orchestra.run.trial038.stdout:The following packages were automatically installed and are no longer required: 2026-02-18T00:39:09.796 INFO:teuthology.orchestra.run.trial038.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-18T00:39:09.796 INFO:teuthology.orchestra.run.trial038.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-18T00:39:09.796 INFO:teuthology.orchestra.run.trial038.stdout: sg3-utils-udev 2026-02-18T00:39:09.796 INFO:teuthology.orchestra.run.trial038.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-18T00:39:09.815 INFO:teuthology.orchestra.run.trial038.stdout:0 upgraded, 0 newly installed, 0 to remove and 53 not upgraded. 2026-02-18T00:39:09.818 DEBUG:teuthology.orchestra.run.trial038:> dpkg -s linux-image-generic 2026-02-18T00:39:09.866 INFO:teuthology.orchestra.run.trial038.stdout:Package: linux-image-generic 2026-02-18T00:39:09.867 INFO:teuthology.orchestra.run.trial038.stdout:Status: install ok installed 2026-02-18T00:39:09.867 INFO:teuthology.orchestra.run.trial038.stdout:Priority: optional 2026-02-18T00:39:09.867 INFO:teuthology.orchestra.run.trial038.stdout:Section: kernel 2026-02-18T00:39:09.867 INFO:teuthology.orchestra.run.trial038.stdout:Installed-Size: 21 2026-02-18T00:39:09.867 INFO:teuthology.orchestra.run.trial038.stdout:Maintainer: Ubuntu Kernel Team 2026-02-18T00:39:09.867 INFO:teuthology.orchestra.run.trial038.stdout:Architecture: amd64 2026-02-18T00:39:09.867 INFO:teuthology.orchestra.run.trial038.stdout:Source: linux-meta 2026-02-18T00:39:09.867 INFO:teuthology.orchestra.run.trial038.stdout:Version: 5.15.0.170.159 2026-02-18T00:39:09.867 INFO:teuthology.orchestra.run.trial038.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.6), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-170), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.6) 2026-02-18T00:39:09.867 INFO:teuthology.orchestra.run.trial038.stdout:Depends: linux-image-5.15.0-170-generic, linux-modules-extra-5.15.0-170-generic, linux-firmware, intel-microcode, amd64-microcode 2026-02-18T00:39:09.867 INFO:teuthology.orchestra.run.trial038.stdout:Recommends: thermald 2026-02-18T00:39:09.868 INFO:teuthology.orchestra.run.trial038.stdout:Description: Generic Linux kernel image 2026-02-18T00:39:09.868 INFO:teuthology.orchestra.run.trial038.stdout: This package will always depend on the latest generic kernel image 2026-02-18T00:39:09.868 INFO:teuthology.orchestra.run.trial038.stdout: available. 2026-02-18T00:39:09.869 DEBUG:teuthology.orchestra.run.trial038:> mktemp 2026-02-18T00:39:09.913 INFO:teuthology.orchestra.run.trial038.stdout:/tmp/tmp.7eqWD5UYcD 2026-02-18T00:39:09.914 DEBUG:teuthology.orchestra.run.trial038:> sudo cp /boot/grub/grub.cfg /tmp/tmp.7eqWD5UYcD 2026-02-18T00:39:09.962 DEBUG:teuthology.orchestra.run.trial038:> sudo chmod 0666 /tmp/tmp.7eqWD5UYcD 2026-02-18T00:39:10.060 DEBUG:teuthology.orchestra.remote:trial038:/tmp/tmp.7eqWD5UYcD is 7KB 2026-02-18T00:39:10.068 DEBUG:teuthology.orchestra.run.trial038:> rm -fr /tmp/tmp.7eqWD5UYcD 2026-02-18T00:39:10.073 DEBUG:teuthology.orchestra.run.trial038:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2026-02-18T00:39:10.125 DEBUG:teuthology.orchestra.run.trial038:> set -ex 2026-02-18T00:39:10.125 DEBUG:teuthology.orchestra.run.trial038:> sudo dd of=/etc/grub.d/01_ceph_kernel 2026-02-18T00:39:10.126 DEBUG:teuthology.orchestra.run.trial038:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2026-02-18T00:39:10.180 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-170-generic 2026-02-18T00:39:10.180 DEBUG:teuthology.orchestra.run.trial038:> sudo update-grub 2026-02-18T00:39:10.415 INFO:teuthology.orchestra.run.trial038.stderr:Sourcing file `/etc/default/grub' 2026-02-18T00:39:10.435 INFO:teuthology.orchestra.run.trial038.stderr:Sourcing file `/etc/default/grub.d/50-curtin-settings.cfg' 2026-02-18T00:39:10.436 INFO:teuthology.orchestra.run.trial038.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2026-02-18T00:39:10.438 INFO:teuthology.orchestra.run.trial038.stderr:Generating grub configuration file ... 2026-02-18T00:39:10.520 INFO:teuthology.orchestra.run.trial038.stderr:Found linux image: /boot/vmlinuz-5.15.0-170-generic 2026-02-18T00:39:10.526 INFO:teuthology.orchestra.run.trial038.stderr:Found initrd image: /boot/initrd.img-5.15.0-170-generic 2026-02-18T00:39:10.701 INFO:teuthology.orchestra.run.trial038.stderr:Found linux image: /boot/vmlinuz-5.15.0-164-generic 2026-02-18T00:39:10.707 INFO:teuthology.orchestra.run.trial038.stderr:Found initrd image: /boot/initrd.img-5.15.0-164-generic 2026-02-18T00:39:10.777 INFO:teuthology.orchestra.run.trial038.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2026-02-18T00:39:10.777 INFO:teuthology.orchestra.run.trial038.stderr:Systems on them will not be added to the GRUB boot configuration. 2026-02-18T00:39:10.777 INFO:teuthology.orchestra.run.trial038.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2026-02-18T00:39:10.780 INFO:teuthology.orchestra.run.trial038.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2026-02-18T00:39:10.787 INFO:teuthology.orchestra.run.trial038.stderr:done 2026-02-18T00:39:10.789 INFO:teuthology.orchestra.remote:Triggering safe hard reboot via SysRq on trial038 2026-02-18T00:39:10.789 DEBUG:teuthology.orchestra.run.trial038:> sudo bash -c ' 2026-02-18T00:39:10.789 DEBUG:teuthology.orchestra.run.trial038:> echo pci > /sys/kernel/reboot/type && echo 1 > /sys/kernel/reboot/force && sync && echo s > /proc/sysrq-trigger && echo u > /proc/sysrq-trigger && sleep 2 && echo b > /proc/sysrq-trigger 2026-02-18T00:39:10.789 DEBUG:teuthology.orchestra.run.trial038:> ' 2026-02-18T00:39:40.791 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2026-02-18T00:39:40.792 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial038.front.sepia.ceph.com' 2026-02-18T00:39:40.793 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-18T00:40:40.844 DEBUG:teuthology.orchestra.remote:timed out 2026-02-18T00:40:49.853 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial038.front.sepia.ceph.com' 2026-02-18T00:40:49.855 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-18T00:40:49.983 DEBUG:teuthology.orchestra.run.trial038:> true 2026-02-18T00:40:50.369 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial038.front.sepia.ceph.com' 2026-02-18T00:40:50.370 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.15.0-170-generic"... 2026-02-18T00:40:50.370 DEBUG:teuthology.orchestra.run.trial038:> uname -r 2026-02-18T00:40:50.415 INFO:teuthology.orchestra.run.trial038.stdout:5.15.0-170-generic 2026-02-18T00:40:50.416 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-170-generic vs 5.15.0-170-generic 2026-02-18T00:40:50.416 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2026-02-18T00:40:50.416 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2026-02-18T00:40:51.417 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2026-02-18T00:40:51.418 DEBUG:teuthology.orchestra.run.trial038:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2026-02-18T00:40:51.430 INFO:teuthology.orchestra.run.trial038.stdout:ttyS1 2026-02-18T00:40:51.458 DEBUG:teuthology.parallel:result is None 2026-02-18T00:40:51.458 INFO:teuthology.run_tasks:Running task internal.base... 2026-02-18T00:40:51.469 INFO:teuthology.task.internal:Creating test directory... 2026-02-18T00:40:51.469 DEBUG:teuthology.orchestra.run.trial038:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2026-02-18T00:40:51.504 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2026-02-18T00:40:51.515 INFO:teuthology.run_tasks:Running task internal.archive... 2026-02-18T00:40:51.529 INFO:teuthology.task.internal:Creating archive directory... 2026-02-18T00:40:51.529 DEBUG:teuthology.orchestra.run.trial038:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2026-02-18T00:40:51.554 INFO:teuthology.run_tasks:Running task internal.coredump... 2026-02-18T00:40:51.566 INFO:teuthology.task.internal:Enabling coredump saving... 2026-02-18T00:40:51.567 DEBUG:teuthology.orchestra.run.trial038:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2026-02-18T00:40:51.600 INFO:teuthology.orchestra.run.trial038.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:40:51.607 INFO:teuthology.orchestra.run.trial038.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:40:51.608 INFO:teuthology.run_tasks:Running task internal.sudo... 2026-02-18T00:40:51.620 INFO:teuthology.task.internal:Configuring sudo... 2026-02-18T00:40:51.621 DEBUG:teuthology.orchestra.run.trial038:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2026-02-18T00:40:51.660 INFO:teuthology.run_tasks:Running task internal.syslog... 2026-02-18T00:40:51.678 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2026-02-18T00:40:51.678 DEBUG:teuthology.orchestra.run.trial038:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2026-02-18T00:40:51.708 DEBUG:teuthology.orchestra.run.trial038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-18T00:40:51.752 DEBUG:teuthology.orchestra.run.trial038:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-18T00:40:51.800 DEBUG:teuthology.orchestra.run.trial038:> set -ex 2026-02-18T00:40:51.800 DEBUG:teuthology.orchestra.run.trial038:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2026-02-18T00:40:51.852 DEBUG:teuthology.orchestra.run.trial038:> sudo service rsyslog restart 2026-02-18T00:40:51.913 INFO:teuthology.run_tasks:Running task internal.timer... 2026-02-18T00:40:51.928 INFO:teuthology.task.internal:Starting timer... 2026-02-18T00:40:51.928 INFO:teuthology.run_tasks:Running task pcp... 2026-02-18T00:40:51.949 INFO:teuthology.run_tasks:Running task selinux... 2026-02-18T00:40:51.969 DEBUG:teuthology.task.selinux:Excluding trial038: OS 'ubuntu' does not support SELinux 2026-02-18T00:40:51.970 DEBUG:teuthology.task.selinux:Getting current SELinux state 2026-02-18T00:40:51.970 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2026-02-18T00:40:51.970 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2026-02-18T00:40:51.970 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2026-02-18T00:40:51.993 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2026-02-18T00:40:51.996 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2026-02-18T00:40:52.134 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2026-02-18T00:40:52.147 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2026-02-18T00:40:52.149 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit trial038.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2026-02-18T00:41:52.065 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@trial038.front.sepia.ceph.com')] 2026-02-18T00:41:52.067 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@trial038.front.sepia.ceph.com' 2026-02-18T00:41:52.069 DEBUG:teuthology.orchestra.connection:{'hostname': 'trial038.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2026-02-18T00:41:52.177 DEBUG:teuthology.orchestra.run.trial038:> true 2026-02-18T00:41:52.429 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@trial038.front.sepia.ceph.com' 2026-02-18T00:41:52.430 INFO:teuthology.run_tasks:Running task clock... 2026-02-18T00:41:52.449 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2026-02-18T00:41:52.450 INFO:teuthology.orchestra.run:Running command with timeout 360 2026-02-18T00:41:52.451 DEBUG:teuthology.orchestra.run.trial038:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-18T00:41:52.490 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2026-02-18T00:41:52.491 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: Command line: ntpd -gq 2026-02-18T00:41:52.491 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: ---------------------------------------------------- 2026-02-18T00:41:52.491 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: ntp-4 is maintained by Network Time Foundation, 2026-02-18T00:41:52.491 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2026-02-18T00:41:52.491 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: corporation. Support and training for ntp-4 are 2026-02-18T00:41:52.491 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: available at https://www.nwtime.org/support 2026-02-18T00:41:52.491 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: ---------------------------------------------------- 2026-02-18T00:41:52.491 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: proto: precision = 0.030 usec (-25) 2026-02-18T00:41:52.491 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: basedate set to 2022-02-04 2026-02-18T00:41:52.491 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: gps base set to 2022-02-06 (week 2196) 2026-02-18T00:41:52.492 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-18T00:41:52.492 INFO:teuthology.orchestra.run.trial038.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2026-02-18T00:41:52.492 INFO:teuthology.orchestra.run.trial038.stderr:restrict ::: KOD does nothing without LIMITED. 2026-02-18T00:41:52.493 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: restrict ::: KOD does nothing without LIMITED. 2026-02-18T00:41:52.493 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: Listen and drop on 0 v6wildcard [::]:123 2026-02-18T00:41:52.493 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2026-02-18T00:41:52.493 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: Listen normally on 2 lo 127.0.0.1:123 2026-02-18T00:41:52.494 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: Listen normally on 3 enp1s0f0 10.20.193.38:123 2026-02-18T00:41:52.494 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: Listen normally on 4 lo [::1]:123 2026-02-18T00:41:52.494 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: Listen normally on 5 enp1s0f0 [fe80::925a:8ff:fe77:5ed4%2]:123 2026-02-18T00:41:52.494 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:41:52 ntpd[4912]: Listening on routing socket on fd #22 for interface updates 2026-02-18T00:41:55.491 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:55 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:41:55.491 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:55 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:41:55.491 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:55 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:41:55.491 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:55 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:41:55.491 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:55 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:41:55.491 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:55 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:41:55.560 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:56 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:41:55.560 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:56 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:41:55.560 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:56 ntpd[4912]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-18T00:41:55.561 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:56 ntpd[4912]: can't open /var/log/ntpstats/peerstats.20260218: Permission denied 2026-02-18T00:41:56.490 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:56 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:41:56.490 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:56 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:41:56.495 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:56 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:41:56.495 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:56 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:41:56.496 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:56 ntpd[4912]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-18T00:41:56.496 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:56 ntpd[4912]: can't open /var/log/ntpstats/peerstats.20260218: Permission denied 2026-02-18T00:41:57.490 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:57 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:41:57.490 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:57 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:41:57.562 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:58 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:41:57.562 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:58 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:41:57.562 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:58 ntpd[4912]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-18T00:41:57.562 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:58 ntpd[4912]: can't open /var/log/ntpstats/peerstats.20260218: Permission denied 2026-02-18T00:41:58.490 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:58 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:41:58.490 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:58 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:41:58.496 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:58 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:41:58.496 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:58 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:41:58.496 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:58 ntpd[4912]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-18T00:41:58.496 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:58 ntpd[4912]: can't open /var/log/ntpstats/peerstats.20260218: Permission denied 2026-02-18T00:41:59.490 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:59 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:41:59.490 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:41:59 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:41:59.560 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:00 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:41:59.560 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:00 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:41:59.561 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:00 ntpd[4912]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-18T00:41:59.561 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:00 ntpd[4912]: can't open /var/log/ntpstats/peerstats.20260218: Permission denied 2026-02-18T00:42:00.490 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:00 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:42:00.490 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:00 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:42:00.498 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:00 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:42:00.498 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:00 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:42:00.498 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:00 ntpd[4912]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-18T00:42:00.498 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:00 ntpd[4912]: can't open /var/log/ntpstats/peerstats.20260218: Permission denied 2026-02-18T00:42:01.490 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:01 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:42:01.490 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:01 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:42:01.563 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:02 ntpd[4912]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2026-02-18T00:42:01.563 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:02 ntpd[4912]: can't open /var/log/ntpstats/rawstats.20260218: Permission denied 2026-02-18T00:42:01.563 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:02 ntpd[4912]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2026-02-18T00:42:01.563 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:02 ntpd[4912]: can't open /var/log/ntpstats/peerstats.20260218: Permission denied 2026-02-18T00:42:01.564 INFO:teuthology.orchestra.run.trial038.stdout:18 Feb 00:42:01 ntpd[4912]: ntpd: time set -0.502430 s 2026-02-18T00:42:01.564 INFO:teuthology.orchestra.run.trial038.stdout:ntpd: time set -0.502430s 2026-02-18T00:42:01.564 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:01 ntpd[4912]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2026-02-18T00:42:01.565 INFO:teuthology.orchestra.run.trial038.stderr:18 Feb 00:42:01 ntpd[4912]: can't open /var/log/ntpstats/loopstats.20260218: Permission denied 2026-02-18T00:42:01.642 INFO:teuthology.orchestra.run.trial038.stdout: remote refid st t when poll reach delay offset jitter 2026-02-18T00:42:01.642 INFO:teuthology.orchestra.run.trial038.stdout:============================================================================== 2026-02-18T00:42:01.642 INFO:teuthology.orchestra.run.trial038.stdout: blotch.image1te .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-18T00:42:01.642 INFO:teuthology.orchestra.run.trial038.stdout: 216.144.228.179 .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-18T00:42:01.642 INFO:teuthology.orchestra.run.trial038.stdout: s2-b.time.mci1. .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-18T00:42:01.643 INFO:teuthology.orchestra.run.trial038.stdout: ntp2.zeus.frume .INIT. 16 u - 64 0 0.000 +0.000 0.000 2026-02-18T00:42:01.644 INFO:teuthology.run_tasks:Running task install... 2026-02-18T00:42:01.658 DEBUG:teuthology.task.install:project ceph 2026-02-18T00:42:01.658 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'}} 2026-02-18T00:42:01.658 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': 'adc1923395f2bcd6ae89b5d331d278364d1848ed'} 2026-02-18T00:42:01.658 INFO:teuthology.task.install:Using flavor: default 2026-02-18T00:42:01.667 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2026-02-18T00:42:01.667 INFO:teuthology.task.install:extra packages: [] 2026-02-18T00:42:01.667 DEBUG:teuthology.orchestra.run.trial038:> sudo apt-key list | grep Ceph 2026-02-18T00:42:01.735 INFO:teuthology.orchestra.run.trial038.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2026-02-18T00:42:01.762 INFO:teuthology.orchestra.run.trial038.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2026-02-18T00:42:01.762 INFO:teuthology.orchestra.run.trial038.stdout:uid [ unknown] Ceph.com (release key) 2026-02-18T00:42:01.763 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2026-02-18T00:42:01.764 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=adc1923395f2bcd6ae89b5d331d278364d1848ed 2026-02-18T00:42:01.900 INFO:teuthology.task.install.deb:Pulling from https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default/ 2026-02-18T00:42:01.901 INFO:teuthology.task.install.deb:Package version is 20.2.0-670-gadc19233-1jammy 2026-02-18T00:42:02.038 DEBUG:teuthology.orchestra.run.trial038:> set -ex 2026-02-18T00:42:02.038 DEBUG:teuthology.orchestra.run.trial038:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2026-02-18T00:42:02.048 DEBUG:teuthology.orchestra.run.trial038:> sudo apt-get update 2026-02-18T00:42:02.209 INFO:teuthology.orchestra.run.trial038.stdout:Ign:1 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy InRelease 2026-02-18T00:42:02.221 INFO:teuthology.orchestra.run.trial038.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy Release [7680 B] 2026-02-18T00:42:02.233 INFO:teuthology.orchestra.run.trial038.stdout:Ign:3 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy Release.gpg 2026-02-18T00:42:02.244 INFO:teuthology.orchestra.run.trial038.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 Packages [18.8 kB] 2026-02-18T00:42:02.316 INFO:teuthology.orchestra.run.trial038.stdout:Hit:5 https://archive.ubuntu.com/ubuntu jammy InRelease 2026-02-18T00:42:02.320 INFO:teuthology.orchestra.run.trial038.stdout:Hit:6 https://archive.ubuntu.com/ubuntu jammy-updates InRelease 2026-02-18T00:42:02.330 INFO:teuthology.orchestra.run.trial038.stdout:Hit:7 https://archive.ubuntu.com/ubuntu jammy-security InRelease 2026-02-18T00:42:02.340 INFO:teuthology.orchestra.run.trial038.stdout:Hit:8 https://archive.ubuntu.com/ubuntu jammy-backports InRelease 2026-02-18T00:42:02.653 INFO:teuthology.orchestra.run.trial038.stdout:Fetched 26.5 kB in 0s (61.2 kB/s) 2026-02-18T00:42:03.162 INFO:teuthology.orchestra.run.trial038.stdout:Reading package lists... 2026-02-18T00:42:03.175 DEBUG:teuthology.orchestra.run.trial038:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=20.2.0-670-gadc19233-1jammy cephadm=20.2.0-670-gadc19233-1jammy ceph-mds=20.2.0-670-gadc19233-1jammy ceph-mgr=20.2.0-670-gadc19233-1jammy ceph-common=20.2.0-670-gadc19233-1jammy ceph-fuse=20.2.0-670-gadc19233-1jammy ceph-test=20.2.0-670-gadc19233-1jammy ceph-volume=20.2.0-670-gadc19233-1jammy radosgw=20.2.0-670-gadc19233-1jammy python3-rados=20.2.0-670-gadc19233-1jammy python3-rgw=20.2.0-670-gadc19233-1jammy python3-cephfs=20.2.0-670-gadc19233-1jammy python3-rbd=20.2.0-670-gadc19233-1jammy libcephfs2=20.2.0-670-gadc19233-1jammy libcephfs-dev=20.2.0-670-gadc19233-1jammy librados2=20.2.0-670-gadc19233-1jammy librbd1=20.2.0-670-gadc19233-1jammy rbd-fuse=20.2.0-670-gadc19233-1jammy 2026-02-18T00:42:03.209 INFO:teuthology.orchestra.run.trial038.stdout:Reading package lists... 2026-02-18T00:42:03.318 INFO:teuthology.orchestra.run.trial038.stdout:Building dependency tree... 2026-02-18T00:42:03.319 INFO:teuthology.orchestra.run.trial038.stdout:Reading state information... 2026-02-18T00:42:03.408 INFO:teuthology.orchestra.run.trial038.stdout:The following packages were automatically installed and are no longer required: 2026-02-18T00:42:03.408 INFO:teuthology.orchestra.run.trial038.stdout: kpartx libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 2026-02-18T00:42:03.408 INFO:teuthology.orchestra.run.trial038.stdout: libgfxdr0 libglusterfs0 libiscsi7 libpmemobj1 libsgutils2-2 sg3-utils 2026-02-18T00:42:03.408 INFO:teuthology.orchestra.run.trial038.stdout: sg3-utils-udev 2026-02-18T00:42:03.408 INFO:teuthology.orchestra.run.trial038.stdout:Use 'sudo apt autoremove' to remove them. 2026-02-18T00:42:03.408 INFO:teuthology.orchestra.run.trial038.stdout:The following additional packages will be installed: 2026-02-18T00:42:03.408 INFO:teuthology.orchestra.run.trial038.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2026-02-18T00:42:03.409 INFO:teuthology.orchestra.run.trial038.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2026-02-18T00:42:03.409 INFO:teuthology.orchestra.run.trial038.stdout: libcephfs-daemon libcephfs-proxy2 libdouble-conversion3 libfuse2 libjq1 2026-02-18T00:42:03.409 INFO:teuthology.orchestra.run.trial038.stdout: liblttng-ust1 libnbd0 liboath0 libonig5 libpcre2-16-0 libqt5core5a 2026-02-18T00:42:03.409 INFO:teuthology.orchestra.run.trial038.stdout: libqt5dbus5 libqt5network5 libradosstriper1 librdkafka1 librgw2 2026-02-18T00:42:03.409 INFO:teuthology.orchestra.run.trial038.stdout: libsqlite3-mod-ceph libthrift-0.16.0 nvme-cli python-asyncssh-doc 2026-02-18T00:42:03.409 INFO:teuthology.orchestra.run.trial038.stdout: python3-asyncssh python3-cachetools python3-ceph-argparse 2026-02-18T00:42:03.409 INFO:teuthology.orchestra.run.trial038.stdout: python3-ceph-common python3-cheroot python3-cherrypy3 python3-google-auth 2026-02-18T00:42:03.409 INFO:teuthology.orchestra.run.trial038.stdout: python3-iniconfig python3-jaraco.classes python3-jaraco.collections 2026-02-18T00:42:03.409 INFO:teuthology.orchestra.run.trial038.stdout: python3-jaraco.functools python3-jaraco.text python3-joblib 2026-02-18T00:42:03.409 INFO:teuthology.orchestra.run.trial038.stdout: python3-kubernetes python3-natsort python3-pluggy python3-portend 2026-02-18T00:42:03.409 INFO:teuthology.orchestra.run.trial038.stdout: python3-prettytable python3-psutil python3-py python3-pygments 2026-02-18T00:42:03.409 INFO:teuthology.orchestra.run.trial038.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2026-02-18T00:42:03.409 INFO:teuthology.orchestra.run.trial038.stdout: python3-rsa python3-simplejson python3-sklearn python3-sklearn-lib 2026-02-18T00:42:03.410 INFO:teuthology.orchestra.run.trial038.stdout: python3-tempora python3-threadpoolctl python3-toml python3-wcwidth 2026-02-18T00:42:03.410 INFO:teuthology.orchestra.run.trial038.stdout: python3-webob python3-websocket python3-zc.lockfile qttranslations5-l10n 2026-02-18T00:42:03.410 INFO:teuthology.orchestra.run.trial038.stdout: socat xmlstarlet 2026-02-18T00:42:03.410 INFO:teuthology.orchestra.run.trial038.stdout:Suggested packages: 2026-02-18T00:42:03.410 INFO:teuthology.orchestra.run.trial038.stdout: python3-influxdb liblua5.3-dev luarocks python-natsort-doc python-psutil-doc 2026-02-18T00:42:03.410 INFO:teuthology.orchestra.run.trial038.stdout: subversion python-pygments-doc ttf-bitstream-vera python3-paste python3-dap 2026-02-18T00:42:03.410 INFO:teuthology.orchestra.run.trial038.stdout: python-sklearn-doc ipython3 python-webob-doc 2026-02-18T00:42:03.410 INFO:teuthology.orchestra.run.trial038.stdout:Recommended packages: 2026-02-18T00:42:03.410 INFO:teuthology.orchestra.run.trial038.stdout: btrfs-tools 2026-02-18T00:42:03.442 INFO:teuthology.orchestra.run.trial038.stdout:The following NEW packages will be installed: 2026-02-18T00:42:03.442 INFO:teuthology.orchestra.run.trial038.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2026-02-18T00:42:03.442 INFO:teuthology.orchestra.run.trial038.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2026-02-18T00:42:03.442 INFO:teuthology.orchestra.run.trial038.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2026-02-18T00:42:03.443 INFO:teuthology.orchestra.run.trial038.stdout: libcephfs-daemon libcephfs-dev libcephfs-proxy2 libcephfs2 2026-02-18T00:42:03.443 INFO:teuthology.orchestra.run.trial038.stdout: libdouble-conversion3 libfuse2 libjq1 liblttng-ust1 libnbd0 liboath0 2026-02-18T00:42:03.443 INFO:teuthology.orchestra.run.trial038.stdout: libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 librados2 2026-02-18T00:42:03.443 INFO:teuthology.orchestra.run.trial038.stdout: libradosstriper1 librbd1 librdkafka1 librgw2 libsqlite3-mod-ceph 2026-02-18T00:42:03.444 INFO:teuthology.orchestra.run.trial038.stdout: libthrift-0.16.0 nvme-cli python-asyncssh-doc python3-asyncssh 2026-02-18T00:42:03.445 INFO:teuthology.orchestra.run.trial038.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common python3-cephfs 2026-02-18T00:42:03.445 INFO:teuthology.orchestra.run.trial038.stdout: python3-cheroot python3-cherrypy3 python3-google-auth python3-iniconfig 2026-02-18T00:42:03.445 INFO:teuthology.orchestra.run.trial038.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2026-02-18T00:42:03.445 INFO:teuthology.orchestra.run.trial038.stdout: python3-jaraco.text python3-joblib python3-kubernetes python3-natsort 2026-02-18T00:42:03.445 INFO:teuthology.orchestra.run.trial038.stdout: python3-pluggy python3-portend python3-prettytable python3-psutil python3-py 2026-02-18T00:42:03.445 INFO:teuthology.orchestra.run.trial038.stdout: python3-pygments python3-pytest python3-rados python3-rbd python3-repoze.lru 2026-02-18T00:42:03.445 INFO:teuthology.orchestra.run.trial038.stdout: python3-requests-oauthlib python3-rgw python3-routes python3-rsa 2026-02-18T00:42:03.445 INFO:teuthology.orchestra.run.trial038.stdout: python3-simplejson python3-sklearn python3-sklearn-lib python3-tempora 2026-02-18T00:42:03.445 INFO:teuthology.orchestra.run.trial038.stdout: python3-threadpoolctl python3-toml python3-wcwidth python3-webob 2026-02-18T00:42:03.445 INFO:teuthology.orchestra.run.trial038.stdout: python3-websocket python3-zc.lockfile qttranslations5-l10n radosgw rbd-fuse 2026-02-18T00:42:03.445 INFO:teuthology.orchestra.run.trial038.stdout: socat xmlstarlet 2026-02-18T00:42:03.523 INFO:teuthology.orchestra.run.trial038.stdout:0 upgraded, 86 newly installed, 0 to remove and 53 not upgraded. 2026-02-18T00:42:03.523 INFO:teuthology.orchestra.run.trial038.stdout:Need to get 280 MB of archives. 2026-02-18T00:42:03.523 INFO:teuthology.orchestra.run.trial038.stdout:After this operation, 1119 MB of additional disk space will be used. 2026-02-18T00:42:03.523 INFO:teuthology.orchestra.run.trial038.stdout:Get:1 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 20.2.0-670-gadc19233-1jammy [3561 kB] 2026-02-18T00:42:03.623 INFO:teuthology.orchestra.run.trial038.stdout:Get:2 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 20.2.0-670-gadc19233-1jammy [2867 kB] 2026-02-18T00:42:03.661 INFO:teuthology.orchestra.run.trial038.stdout:Get:3 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 20.2.0-670-gadc19233-1jammy [833 kB] 2026-02-18T00:42:03.674 INFO:teuthology.orchestra.run.trial038.stdout:Get:4 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 20.2.0-670-gadc19233-1jammy [364 kB] 2026-02-18T00:42:03.680 INFO:teuthology.orchestra.run.trial038.stdout:Get:5 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 20.2.0-670-gadc19233-1jammy [32.8 kB] 2026-02-18T00:42:03.681 INFO:teuthology.orchestra.run.trial038.stdout:Get:6 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 20.2.0-670-gadc19233-1jammy [184 kB] 2026-02-18T00:42:03.684 INFO:teuthology.orchestra.run.trial038.stdout:Get:7 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 20.2.0-670-gadc19233-1jammy [83.9 kB] 2026-02-18T00:42:03.686 INFO:teuthology.orchestra.run.trial038.stdout:Get:8 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 20.2.0-670-gadc19233-1jammy [341 kB] 2026-02-18T00:42:03.691 INFO:teuthology.orchestra.run.trial038.stdout:Get:9 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 20.2.0-670-gadc19233-1jammy [8687 kB] 2026-02-18T00:42:03.834 INFO:teuthology.orchestra.run.trial038.stdout:Get:10 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 20.2.0-670-gadc19233-1jammy [112 kB] 2026-02-18T00:42:03.835 INFO:teuthology.orchestra.run.trial038.stdout:Get:11 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 20.2.0-670-gadc19233-1jammy [261 kB] 2026-02-18T00:42:03.839 INFO:teuthology.orchestra.run.trial038.stdout:Get:12 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 20.2.0-670-gadc19233-1jammy [29.3 MB] 2026-02-18T00:42:03.876 INFO:teuthology.orchestra.run.trial038.stdout:Get:13 https://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2026-02-18T00:42:04.307 INFO:teuthology.orchestra.run.trial038.stdout:Get:14 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2026-02-18T00:42:04.321 INFO:teuthology.orchestra.run.trial038.stdout:Get:15 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2026-02-18T00:42:04.324 INFO:teuthology.orchestra.run.trial038.stdout:Get:16 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 20.2.0-670-gadc19233-1jammy [5416 kB] 2026-02-18T00:42:04.409 INFO:teuthology.orchestra.run.trial038.stdout:Get:17 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 20.2.0-670-gadc19233-1jammy [246 kB] 2026-02-18T00:42:04.409 INFO:teuthology.orchestra.run.trial038.stdout:Get:18 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2006 kB] 2026-02-18T00:42:04.412 INFO:teuthology.orchestra.run.trial038.stdout:Get:19 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 20.2.0-670-gadc19233-1jammy [124 kB] 2026-02-18T00:42:04.414 INFO:teuthology.orchestra.run.trial038.stdout:Get:20 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 20.2.0-670-gadc19233-1jammy [907 kB] 2026-02-18T00:42:04.429 INFO:teuthology.orchestra.run.trial038.stdout:Get:21 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 20.2.0-670-gadc19233-1jammy [6397 kB] 2026-02-18T00:42:04.535 INFO:teuthology.orchestra.run.trial038.stdout:Get:22 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 20.2.0-670-gadc19233-1jammy [21.7 MB] 2026-02-18T00:42:04.664 INFO:teuthology.orchestra.run.trial038.stdout:Get:23 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2026-02-18T00:42:04.677 INFO:teuthology.orchestra.run.trial038.stdout:Get:24 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2026-02-18T00:42:04.711 INFO:teuthology.orchestra.run.trial038.stdout:Get:25 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2026-02-18T00:42:04.721 INFO:teuthology.orchestra.run.trial038.stdout:Get:26 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libnbd0 amd64 1.10.5-1 [71.3 kB] 2026-02-18T00:42:04.724 INFO:teuthology.orchestra.run.trial038.stdout:Get:27 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2026-02-18T00:42:04.725 INFO:teuthology.orchestra.run.trial038.stdout:Get:28 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2026-02-18T00:42:04.726 INFO:teuthology.orchestra.run.trial038.stdout:Get:29 https://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2026-02-18T00:42:04.747 INFO:teuthology.orchestra.run.trial038.stdout:Get:30 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 liboath0 amd64 2.6.7-3ubuntu0.1 [41.3 kB] 2026-02-18T00:42:04.748 INFO:teuthology.orchestra.run.trial038.stdout:Get:31 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9030 B] 2026-02-18T00:42:04.756 INFO:teuthology.orchestra.run.trial038.stdout:Get:32 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2026-02-18T00:42:04.840 INFO:teuthology.orchestra.run.trial038.stdout:Get:33 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6452 B] 2026-02-18T00:42:04.840 INFO:teuthology.orchestra.run.trial038.stdout:Get:34 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8716 B] 2026-02-18T00:42:04.840 INFO:teuthology.orchestra.run.trial038.stdout:Get:35 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2026-02-18T00:42:04.841 INFO:teuthology.orchestra.run.trial038.stdout:Get:36 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2026-02-18T00:42:04.841 INFO:teuthology.orchestra.run.trial038.stdout:Get:37 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7240 B] 2026-02-18T00:42:04.841 INFO:teuthology.orchestra.run.trial038.stdout:Get:38 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8980 B] 2026-02-18T00:42:04.841 INFO:teuthology.orchestra.run.trial038.stdout:Get:39 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2026-02-18T00:42:04.847 INFO:teuthology.orchestra.run.trial038.stdout:Get:40 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2026-02-18T00:42:04.848 INFO:teuthology.orchestra.run.trial038.stdout:Get:41 https://archive.ubuntu.com/ubuntu jammy/universe amd64 libfuse2 amd64 2.9.9-5ubuntu3 [90.3 kB] 2026-02-18T00:42:04.892 INFO:teuthology.orchestra.run.trial038.stdout:Get:42 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 20.2.0-670-gadc19233-1jammy [14.1 kB] 2026-02-18T00:42:04.892 INFO:teuthology.orchestra.run.trial038.stdout:Get:43 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 20.2.0-670-gadc19233-1jammy [957 kB] 2026-02-18T00:42:04.906 INFO:teuthology.orchestra.run.trial038.stdout:Get:44 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 20.2.0-670-gadc19233-1jammy [2345 kB] 2026-02-18T00:42:04.935 INFO:teuthology.orchestra.run.trial038.stdout:Get:45 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python3-asyncssh all 2.5.0-1ubuntu0.1 [189 kB] 2026-02-18T00:42:04.938 INFO:teuthology.orchestra.run.trial038.stdout:Get:46 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2026-02-18T00:42:04.938 INFO:teuthology.orchestra.run.trial038.stdout:Get:47 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2026-02-18T00:42:04.939 INFO:teuthology.orchestra.run.trial038.stdout:Get:48 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2058 kB] 2026-02-18T00:42:04.944 INFO:teuthology.orchestra.run.trial038.stdout:Get:49 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 20.2.0-670-gadc19233-1jammy [1049 kB] 2026-02-18T00:42:04.961 INFO:teuthology.orchestra.run.trial038.stdout:Get:50 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 20.2.0-670-gadc19233-1jammy [179 kB] 2026-02-18T00:42:04.963 INFO:teuthology.orchestra.run.trial038.stdout:Get:51 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 20.2.0-670-gadc19233-1jammy [45.5 MB] 2026-02-18T00:42:04.970 INFO:teuthology.orchestra.run.trial038.stdout:Get:52 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2026-02-18T00:42:04.973 INFO:teuthology.orchestra.run.trial038.stdout:Get:53 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2026-02-18T00:42:04.973 INFO:teuthology.orchestra.run.trial038.stdout:Get:54 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1829 kB] 2026-02-18T00:42:05.046 INFO:teuthology.orchestra.run.trial038.stdout:Get:55 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9722 B] 2026-02-18T00:42:05.047 INFO:teuthology.orchestra.run.trial038.stdout:Get:56 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2026-02-18T00:42:05.047 INFO:teuthology.orchestra.run.trial038.stdout:Get:57 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2026-02-18T00:42:05.048 INFO:teuthology.orchestra.run.trial038.stdout:Get:58 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2026-02-18T00:42:05.048 INFO:teuthology.orchestra.run.trial038.stdout:Get:59 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2026-02-18T00:42:05.048 INFO:teuthology.orchestra.run.trial038.stdout:Get:60 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2026-02-18T00:42:05.124 INFO:teuthology.orchestra.run.trial038.stdout:Get:61 https://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2026-02-18T00:42:05.126 INFO:teuthology.orchestra.run.trial038.stdout:Get:62 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libjq1 amd64 1.6-2.1ubuntu3.1 [133 kB] 2026-02-18T00:42:05.127 INFO:teuthology.orchestra.run.trial038.stdout:Get:63 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 jq amd64 1.6-2.1ubuntu3.1 [52.5 kB] 2026-02-18T00:42:05.216 INFO:teuthology.orchestra.run.trial038.stdout:Get:64 https://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2026-02-18T00:42:05.219 INFO:teuthology.orchestra.run.trial038.stdout:Get:65 https://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2026-02-18T00:42:05.222 INFO:teuthology.orchestra.run.trial038.stdout:Get:66 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.3 [474 kB] 2026-02-18T00:42:05.226 INFO:teuthology.orchestra.run.trial038.stdout:Get:67 https://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 python-asyncssh-doc all 2.5.0-1ubuntu0.1 [309 kB] 2026-02-18T00:42:05.230 INFO:teuthology.orchestra.run.trial038.stdout:Get:68 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6024 B] 2026-02-18T00:42:05.230 INFO:teuthology.orchestra.run.trial038.stdout:Get:69 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2026-02-18T00:42:05.230 INFO:teuthology.orchestra.run.trial038.stdout:Get:70 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2026-02-18T00:42:05.307 INFO:teuthology.orchestra.run.trial038.stdout:Get:71 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2026-02-18T00:42:05.308 INFO:teuthology.orchestra.run.trial038.stdout:Get:72 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-pygments all 2.11.2+dfsg-2ubuntu0.1 [750 kB] 2026-02-18T00:42:05.315 INFO:teuthology.orchestra.run.trial038.stdout:Get:73 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2026-02-18T00:42:05.399 INFO:teuthology.orchestra.run.trial038.stdout:Get:74 https://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2026-02-18T00:42:05.401 INFO:teuthology.orchestra.run.trial038.stdout:Get:75 https://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2026-02-18T00:42:05.401 INFO:teuthology.orchestra.run.trial038.stdout:Get:76 https://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2026-02-18T00:42:05.402 INFO:teuthology.orchestra.run.trial038.stdout:Get:77 https://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1983 kB] 2026-02-18T00:42:05.716 INFO:teuthology.orchestra.run.trial038.stdout:Get:78 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 20.2.0-670-gadc19233-1jammy [8625 kB] 2026-02-18T00:42:05.851 INFO:teuthology.orchestra.run.trial038.stdout:Get:79 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 20.2.0-670-gadc19233-1jammy [14.2 kB] 2026-02-18T00:42:05.851 INFO:teuthology.orchestra.run.trial038.stdout:Get:80 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 20.2.0-670-gadc19233-1jammy [99.5 MB] 2026-02-18T00:42:07.494 INFO:teuthology.orchestra.run.trial038.stdout:Get:81 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 20.2.0-670-gadc19233-1jammy [135 kB] 2026-02-18T00:42:07.495 INFO:teuthology.orchestra.run.trial038.stdout:Get:82 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-daemon amd64 20.2.0-670-gadc19233-1jammy [43.3 kB] 2026-02-18T00:42:07.495 INFO:teuthology.orchestra.run.trial038.stdout:Get:83 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-proxy2 amd64 20.2.0-670-gadc19233-1jammy [30.7 kB] 2026-02-18T00:42:07.496 INFO:teuthology.orchestra.run.trial038.stdout:Get:84 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 20.2.0-670-gadc19233-1jammy [41.5 kB] 2026-02-18T00:42:07.496 INFO:teuthology.orchestra.run.trial038.stdout:Get:85 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 20.2.0-670-gadc19233-1jammy [25.1 MB] 2026-02-18T00:42:07.897 INFO:teuthology.orchestra.run.trial038.stdout:Get:86 https://2.chacra.ceph.com/r/ceph/20.2.1_2.17.26-baseline_2/adc1923395f2bcd6ae89b5d331d278364d1848ed/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 20.2.0-670-gadc19233-1jammy [97.5 kB] 2026-02-18T00:42:08.175 INFO:teuthology.orchestra.run.trial038.stdout:Fetched 280 MB in 4s (62.9 MB/s) 2026-02-18T00:42:08.192 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2026-02-18T00:42:08.347 INFO:teuthology.orchestra.run.trial038.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 167438 files and directories currently installed.) 2026-02-18T00:42:08.349 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../00-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2026-02-18T00:42:08.350 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-18T00:42:08.370 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2026-02-18T00:42:08.383 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../01-libdouble-conversion3_3.1.7-4_amd64.deb ... 2026-02-18T00:42:08.383 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-18T00:42:08.400 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2026-02-18T00:42:08.413 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../02-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2026-02-18T00:42:08.413 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-18T00:42:08.430 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libqt5core5a:amd64. 2026-02-18T00:42:08.441 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../03-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-18T00:42:08.446 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-18T00:42:08.478 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2026-02-18T00:42:08.486 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../04-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-18T00:42:08.487 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-18T00:42:08.503 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libqt5network5:amd64. 2026-02-18T00:42:08.511 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../05-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2026-02-18T00:42:08.512 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-18T00:42:08.531 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2026-02-18T00:42:08.539 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../06-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2026-02-18T00:42:08.539 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-18T00:42:08.556 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package librados2. 2026-02-18T00:42:08.564 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../07-librados2_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:08.565 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking librados2 (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:08.606 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libnbd0. 2026-02-18T00:42:08.614 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../08-libnbd0_1.10.5-1_amd64.deb ... 2026-02-18T00:42:08.614 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libnbd0 (1.10.5-1) ... 2026-02-18T00:42:08.629 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package librbd1. 2026-02-18T00:42:08.637 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../09-librbd1_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:08.637 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking librbd1 (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:08.674 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libcephfs2. 2026-02-18T00:42:08.682 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../10-libcephfs2_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:08.683 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libcephfs2 (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:08.702 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-rados. 2026-02-18T00:42:08.710 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../11-python3-rados_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:08.711 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-rados (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:08.727 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-ceph-argparse. 2026-02-18T00:42:08.735 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../12-python3-ceph-argparse_20.2.0-670-gadc19233-1jammy_all.deb ... 2026-02-18T00:42:08.735 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-ceph-argparse (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:08.750 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-cephfs. 2026-02-18T00:42:08.758 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../13-python3-cephfs_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:08.759 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-cephfs (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:08.775 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-ceph-common. 2026-02-18T00:42:08.783 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../14-python3-ceph-common_20.2.0-670-gadc19233-1jammy_all.deb ... 2026-02-18T00:42:08.783 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-ceph-common (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:08.801 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-wcwidth. 2026-02-18T00:42:08.809 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../15-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2026-02-18T00:42:08.809 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-18T00:42:08.826 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-prettytable. 2026-02-18T00:42:08.834 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../16-python3-prettytable_2.5.0-2_all.deb ... 2026-02-18T00:42:08.834 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2026-02-18T00:42:08.848 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-rbd. 2026-02-18T00:42:08.856 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../17-python3-rbd_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:08.857 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-rbd (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:08.873 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package librdkafka1:amd64. 2026-02-18T00:42:08.881 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../18-librdkafka1_1.8.0-1build1_amd64.deb ... 2026-02-18T00:42:08.882 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-18T00:42:08.901 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package librgw2. 2026-02-18T00:42:08.909 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../19-librgw2_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:08.909 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking librgw2 (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:09.004 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-rgw. 2026-02-18T00:42:09.012 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../20-python3-rgw_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:09.013 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-rgw (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:09.027 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package liboath0:amd64. 2026-02-18T00:42:09.035 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../21-liboath0_2.6.7-3ubuntu0.1_amd64.deb ... 2026-02-18T00:42:09.036 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-18T00:42:09.050 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libradosstriper1. 2026-02-18T00:42:09.058 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../22-libradosstriper1_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:09.058 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libradosstriper1 (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:09.075 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-common. 2026-02-18T00:42:09.083 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../23-ceph-common_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:09.084 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-common (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:09.337 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-base. 2026-02-18T00:42:09.346 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../24-ceph-base_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:09.351 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-base (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:09.414 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-jaraco.functools. 2026-02-18T00:42:09.423 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../25-python3-jaraco.functools_3.4.0-2_all.deb ... 2026-02-18T00:42:09.423 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2026-02-18T00:42:09.438 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-cheroot. 2026-02-18T00:42:09.446 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../26-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2026-02-18T00:42:09.446 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-18T00:42:09.463 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-jaraco.classes. 2026-02-18T00:42:09.471 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../27-python3-jaraco.classes_3.2.1-3_all.deb ... 2026-02-18T00:42:09.472 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2026-02-18T00:42:09.486 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-jaraco.text. 2026-02-18T00:42:09.495 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../28-python3-jaraco.text_3.6.0-2_all.deb ... 2026-02-18T00:42:09.495 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2026-02-18T00:42:09.509 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-jaraco.collections. 2026-02-18T00:42:09.517 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../29-python3-jaraco.collections_3.4.0-2_all.deb ... 2026-02-18T00:42:09.518 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2026-02-18T00:42:09.532 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-tempora. 2026-02-18T00:42:09.541 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../30-python3-tempora_4.1.2-1_all.deb ... 2026-02-18T00:42:09.541 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-tempora (4.1.2-1) ... 2026-02-18T00:42:09.556 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-portend. 2026-02-18T00:42:09.564 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../31-python3-portend_3.0.0-1_all.deb ... 2026-02-18T00:42:09.565 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-portend (3.0.0-1) ... 2026-02-18T00:42:09.579 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-zc.lockfile. 2026-02-18T00:42:09.587 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../32-python3-zc.lockfile_2.0-1_all.deb ... 2026-02-18T00:42:09.588 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2026-02-18T00:42:09.602 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-cherrypy3. 2026-02-18T00:42:09.610 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../33-python3-cherrypy3_18.6.1-4_all.deb ... 2026-02-18T00:42:09.610 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2026-02-18T00:42:09.634 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-natsort. 2026-02-18T00:42:09.643 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../34-python3-natsort_8.0.2-1_all.deb ... 2026-02-18T00:42:09.643 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-natsort (8.0.2-1) ... 2026-02-18T00:42:09.659 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2026-02-18T00:42:09.667 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../35-ceph-mgr-modules-core_20.2.0-670-gadc19233-1jammy_all.deb ... 2026-02-18T00:42:09.667 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-mgr-modules-core (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:09.694 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2026-02-18T00:42:09.702 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../36-libsqlite3-mod-ceph_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:09.703 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libsqlite3-mod-ceph (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:09.718 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-mgr. 2026-02-18T00:42:09.726 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../37-ceph-mgr_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:09.726 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-mgr (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:09.747 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-mon. 2026-02-18T00:42:09.755 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../38-ceph-mon_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:09.755 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-mon (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:09.814 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libfuse2:amd64. 2026-02-18T00:42:09.822 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../39-libfuse2_2.9.9-5ubuntu3_amd64.deb ... 2026-02-18T00:42:09.822 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-18T00:42:09.839 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-osd. 2026-02-18T00:42:09.847 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../40-ceph-osd_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:09.847 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-osd (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:09.989 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph. 2026-02-18T00:42:09.998 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../41-ceph_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:09.998 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:10.011 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-fuse. 2026-02-18T00:42:10.018 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../42-ceph-fuse_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:10.018 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-fuse (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:10.039 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-mds. 2026-02-18T00:42:10.048 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../43-ceph-mds_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:10.048 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-mds (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:10.079 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package cephadm. 2026-02-18T00:42:10.087 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../44-cephadm_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:10.088 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking cephadm (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:10.104 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-asyncssh. 2026-02-18T00:42:10.112 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../45-python3-asyncssh_2.5.0-1ubuntu0.1_all.deb ... 2026-02-18T00:42:10.113 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-18T00:42:10.133 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2026-02-18T00:42:10.141 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../46-ceph-mgr-cephadm_20.2.0-670-gadc19233-1jammy_all.deb ... 2026-02-18T00:42:10.141 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-mgr-cephadm (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:10.162 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-repoze.lru. 2026-02-18T00:42:10.170 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../47-python3-repoze.lru_0.7-2_all.deb ... 2026-02-18T00:42:10.171 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2026-02-18T00:42:10.185 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-routes. 2026-02-18T00:42:10.193 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../48-python3-routes_2.5.1-1ubuntu1_all.deb ... 2026-02-18T00:42:10.194 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2026-02-18T00:42:10.214 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2026-02-18T00:42:10.222 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../49-ceph-mgr-dashboard_20.2.0-670-gadc19233-1jammy_all.deb ... 2026-02-18T00:42:10.222 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-mgr-dashboard (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:10.649 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2026-02-18T00:42:10.657 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../50-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2026-02-18T00:42:10.658 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-18T00:42:10.696 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-joblib. 2026-02-18T00:42:10.705 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../51-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2026-02-18T00:42:10.705 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2026-02-18T00:42:10.732 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-threadpoolctl. 2026-02-18T00:42:10.740 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../52-python3-threadpoolctl_3.1.0-1_all.deb ... 2026-02-18T00:42:10.741 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2026-02-18T00:42:10.755 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-sklearn. 2026-02-18T00:42:10.763 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../53-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2026-02-18T00:42:10.764 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-18T00:42:10.845 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2026-02-18T00:42:10.854 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../54-ceph-mgr-diskprediction-local_20.2.0-670-gadc19233-1jammy_all.deb ... 2026-02-18T00:42:10.854 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-mgr-diskprediction-local (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:11.010 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-cachetools. 2026-02-18T00:42:11.018 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../55-python3-cachetools_5.0.0-1_all.deb ... 2026-02-18T00:42:11.019 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2026-02-18T00:42:11.033 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-rsa. 2026-02-18T00:42:11.041 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../56-python3-rsa_4.8-1_all.deb ... 2026-02-18T00:42:11.041 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-rsa (4.8-1) ... 2026-02-18T00:42:11.059 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-google-auth. 2026-02-18T00:42:11.068 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../57-python3-google-auth_1.5.1-3_all.deb ... 2026-02-18T00:42:11.068 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2026-02-18T00:42:11.086 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-requests-oauthlib. 2026-02-18T00:42:11.094 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../58-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2026-02-18T00:42:11.095 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-18T00:42:11.110 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-websocket. 2026-02-18T00:42:11.119 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../59-python3-websocket_1.2.3-1_all.deb ... 2026-02-18T00:42:11.119 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-websocket (1.2.3-1) ... 2026-02-18T00:42:11.137 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-kubernetes. 2026-02-18T00:42:11.145 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../60-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2026-02-18T00:42:11.145 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-18T00:42:11.241 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2026-02-18T00:42:11.249 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../61-ceph-mgr-k8sevents_20.2.0-670-gadc19233-1jammy_all.deb ... 2026-02-18T00:42:11.250 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-mgr-k8sevents (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:11.264 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libonig5:amd64. 2026-02-18T00:42:11.272 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../62-libonig5_6.9.7.1-2build1_amd64.deb ... 2026-02-18T00:42:11.273 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-18T00:42:11.289 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libjq1:amd64. 2026-02-18T00:42:11.297 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../63-libjq1_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-18T00:42:11.298 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-18T00:42:11.313 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package jq. 2026-02-18T00:42:11.321 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../64-jq_1.6-2.1ubuntu3.1_amd64.deb ... 2026-02-18T00:42:11.321 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking jq (1.6-2.1ubuntu3.1) ... 2026-02-18T00:42:11.335 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package socat. 2026-02-18T00:42:11.342 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../65-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2026-02-18T00:42:11.342 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2026-02-18T00:42:11.363 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package xmlstarlet. 2026-02-18T00:42:11.371 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../66-xmlstarlet_1.6.1-2.1_amd64.deb ... 2026-02-18T00:42:11.371 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2026-02-18T00:42:11.409 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-test. 2026-02-18T00:42:11.417 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../67-ceph-test_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:11.417 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-test (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:12.299 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package ceph-volume. 2026-02-18T00:42:12.308 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../68-ceph-volume_20.2.0-670-gadc19233-1jammy_all.deb ... 2026-02-18T00:42:12.308 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking ceph-volume (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:12.332 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libcephfs-daemon. 2026-02-18T00:42:12.340 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../69-libcephfs-daemon_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:12.340 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libcephfs-daemon (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:12.356 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libcephfs-proxy2. 2026-02-18T00:42:12.364 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../70-libcephfs-proxy2_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:12.365 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libcephfs-proxy2 (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:12.379 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package libcephfs-dev. 2026-02-18T00:42:12.387 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../71-libcephfs-dev_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:12.388 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking libcephfs-dev (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:12.403 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package nvme-cli. 2026-02-18T00:42:12.410 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../72-nvme-cli_1.16-3ubuntu0.3_amd64.deb ... 2026-02-18T00:42:12.410 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking nvme-cli (1.16-3ubuntu0.3) ... 2026-02-18T00:42:12.441 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python-asyncssh-doc. 2026-02-18T00:42:12.449 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../73-python-asyncssh-doc_2.5.0-1ubuntu0.1_all.deb ... 2026-02-18T00:42:12.450 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-18T00:42:12.480 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-iniconfig. 2026-02-18T00:42:12.489 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../74-python3-iniconfig_1.1.1-2_all.deb ... 2026-02-18T00:42:12.489 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2026-02-18T00:42:12.504 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-pluggy. 2026-02-18T00:42:12.512 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../75-python3-pluggy_0.13.0-7.1_all.deb ... 2026-02-18T00:42:12.513 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2026-02-18T00:42:12.529 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-psutil. 2026-02-18T00:42:12.537 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../76-python3-psutil_5.9.0-1build1_amd64.deb ... 2026-02-18T00:42:12.538 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2026-02-18T00:42:12.556 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-py. 2026-02-18T00:42:12.565 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../77-python3-py_1.10.0-1_all.deb ... 2026-02-18T00:42:12.565 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-py (1.10.0-1) ... 2026-02-18T00:42:12.585 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-pygments. 2026-02-18T00:42:12.594 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../78-python3-pygments_2.11.2+dfsg-2ubuntu0.1_all.deb ... 2026-02-18T00:42:12.594 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-18T00:42:12.635 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-toml. 2026-02-18T00:42:12.643 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../79-python3-toml_0.10.2-1_all.deb ... 2026-02-18T00:42:12.644 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-toml (0.10.2-1) ... 2026-02-18T00:42:12.659 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-pytest. 2026-02-18T00:42:12.667 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../80-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2026-02-18T00:42:12.667 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2026-02-18T00:42:12.689 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-simplejson. 2026-02-18T00:42:12.698 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../81-python3-simplejson_3.17.6-1build1_amd64.deb ... 2026-02-18T00:42:12.698 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2026-02-18T00:42:12.716 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package python3-webob. 2026-02-18T00:42:12.725 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../82-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2026-02-18T00:42:12.725 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-18T00:42:12.748 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package qttranslations5-l10n. 2026-02-18T00:42:12.757 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../83-qttranslations5-l10n_5.15.3-1_all.deb ... 2026-02-18T00:42:12.757 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2026-02-18T00:42:12.825 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package radosgw. 2026-02-18T00:42:12.833 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../84-radosgw_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:12.834 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking radosgw (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:13.041 INFO:teuthology.orchestra.run.trial038.stdout:Selecting previously unselected package rbd-fuse. 2026-02-18T00:42:13.049 INFO:teuthology.orchestra.run.trial038.stdout:Preparing to unpack .../85-rbd-fuse_20.2.0-670-gadc19233-1jammy_amd64.deb ... 2026-02-18T00:42:13.050 INFO:teuthology.orchestra.run.trial038.stdout:Unpacking rbd-fuse (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:13.077 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2026-02-18T00:42:13.143 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2026-02-18T00:42:13.144 INFO:teuthology.orchestra.run.trial038.stdout:Setting up nvme-cli (1.16-3ubuntu0.3) ... 2026-02-18T00:42:13.214 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service → /lib/systemd/system/nvmefc-boot-connections.service. 2026-02-18T00:42:13.476 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service → /lib/systemd/system/nvmf-autoconnect.service. 2026-02-18T00:42:13.840 INFO:teuthology.orchestra.run.trial038.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2026-02-18T00:42:13.851 INFO:teuthology.orchestra.run.trial038.stdout:Setting up cephadm (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:13.904 INFO:teuthology.orchestra.run.trial038.stdout:Adding system user cephadm....done 2026-02-18T00:42:13.915 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2026-02-18T00:42:13.989 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python-asyncssh-doc (2.5.0-1ubuntu0.1) ... 2026-02-18T00:42:13.989 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2026-02-18T00:42:14.060 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-repoze.lru (0.7-2) ... 2026-02-18T00:42:14.135 INFO:teuthology.orchestra.run.trial038.stdout:Setting up liboath0:amd64 (2.6.7-3ubuntu0.1) ... 2026-02-18T00:42:14.136 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-py (1.10.0-1) ... 2026-02-18T00:42:14.231 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2026-02-18T00:42:14.356 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-cachetools (5.0.0-1) ... 2026-02-18T00:42:14.431 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2026-02-18T00:42:14.506 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-ceph-argparse (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:14.574 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2026-02-18T00:42:14.575 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libnbd0 (1.10.5-1) ... 2026-02-18T00:42:14.576 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libfuse2:amd64 (2.9.9-5ubuntu3) ... 2026-02-18T00:42:14.577 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2026-02-18T00:42:14.578 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2026-02-18T00:42:14.697 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-natsort (8.0.2-1) ... 2026-02-18T00:42:14.774 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libcephfs-proxy2 (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:14.775 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2026-02-18T00:42:14.850 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2026-02-18T00:42:14.937 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-pygments (2.11.2+dfsg-2ubuntu0.1) ... 2026-02-18T00:42:15.185 INFO:teuthology.orchestra.run.trial038.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2026-02-18T00:42:15.186 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2026-02-18T00:42:15.282 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-asyncssh (2.5.0-1ubuntu0.1) ... 2026-02-18T00:42:15.408 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2026-02-18T00:42:15.489 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2026-02-18T00:42:15.561 INFO:teuthology.orchestra.run.trial038.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2026-02-18T00:42:15.562 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-ceph-common (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:15.649 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2026-02-18T00:42:16.143 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-18T00:42:16.147 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-toml (0.10.2-1) ... 2026-02-18T00:42:16.223 INFO:teuthology.orchestra.run.trial038.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2026-02-18T00:42:16.224 INFO:teuthology.orchestra.run.trial038.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2026-02-18T00:42:16.225 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2026-02-18T00:42:16.299 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2026-02-18T00:42:16.372 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-18T00:42:16.373 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-rsa (4.8-1) ... 2026-02-18T00:42:16.450 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-tempora (4.1.2-1) ... 2026-02-18T00:42:16.522 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-prettytable (2.5.0-2) ... 2026-02-18T00:42:16.602 INFO:teuthology.orchestra.run.trial038.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2026-02-18T00:42:16.603 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-websocket (1.2.3-1) ... 2026-02-18T00:42:16.683 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2026-02-18T00:42:16.684 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2026-02-18T00:42:16.759 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2026-02-18T00:42:16.843 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2026-02-18T00:42:16.917 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3.1) ... 2026-02-18T00:42:16.918 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2026-02-18T00:42:17.054 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-portend (3.0.0-1) ... 2026-02-18T00:42:17.125 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2026-02-18T00:42:17.126 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-google-auth (1.5.1-3) ... 2026-02-18T00:42:17.205 INFO:teuthology.orchestra.run.trial038.stdout:Setting up jq (1.6-2.1ubuntu3.1) ... 2026-02-18T00:42:17.206 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2026-02-18T00:42:17.338 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2026-02-18T00:42:17.339 INFO:teuthology.orchestra.run.trial038.stdout:Setting up librados2 (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:17.340 INFO:teuthology.orchestra.run.trial038.stdout:Setting up librgw2 (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:17.341 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libsqlite3-mod-ceph (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:17.342 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2026-02-18T00:42:17.856 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libcephfs2 (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:17.857 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libradosstriper1 (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:17.858 INFO:teuthology.orchestra.run.trial038.stdout:Setting up librbd1 (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:17.859 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-mgr-modules-core (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:17.860 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-fuse (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:17.931 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-18T00:42:17.932 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target → /lib/systemd/system/ceph-fuse.target. 2026-02-18T00:42:18.326 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libcephfs-dev (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:18.326 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-rados (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:18.327 INFO:teuthology.orchestra.run.trial038.stdout:Setting up libcephfs-daemon (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:18.328 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-rbd (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:18.329 INFO:teuthology.orchestra.run.trial038.stdout:Setting up rbd-fuse (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:18.330 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-rgw (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:18.331 INFO:teuthology.orchestra.run.trial038.stdout:Setting up python3-cephfs (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:18.332 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-common (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:18.358 INFO:teuthology.orchestra.run.trial038.stdout:Adding group ceph....done 2026-02-18T00:42:18.393 INFO:teuthology.orchestra.run.trial038.stdout:Adding system user ceph....done 2026-02-18T00:42:18.403 INFO:teuthology.orchestra.run.trial038.stdout:Setting system user ceph properties....done 2026-02-18T00:42:18.410 INFO:teuthology.orchestra.run.trial038.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2026-02-18T00:42:18.496 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /lib/systemd/system/ceph.target. 2026-02-18T00:42:18.756 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service → /lib/systemd/system/rbdmap.service. 2026-02-18T00:42:19.113 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-test (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:19.114 INFO:teuthology.orchestra.run.trial038.stdout:Setting up radosgw (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:19.360 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-18T00:42:19.360 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /lib/systemd/system/ceph-radosgw.target. 2026-02-18T00:42:19.702 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-base (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:19.816 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /lib/systemd/system/ceph-crash.service. 2026-02-18T00:42:20.171 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-mds (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:20.230 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-18T00:42:20.231 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /lib/systemd/system/ceph-mds.target. 2026-02-18T00:42:20.583 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-mgr (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:20.662 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-18T00:42:20.662 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /lib/systemd/system/ceph-mgr.target. 2026-02-18T00:42:21.002 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-osd (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:21.091 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-18T00:42:21.091 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /lib/systemd/system/ceph-osd.target. 2026-02-18T00:42:21.439 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-mgr-k8sevents (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:21.448 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-mgr-diskprediction-local (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:21.458 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-mon (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:21.531 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-18T00:42:21.532 INFO:teuthology.orchestra.run.trial038.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /lib/systemd/system/ceph-mon.target. 2026-02-18T00:42:21.890 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-mgr-cephadm (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:21.902 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:21.903 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-mgr-dashboard (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:21.915 INFO:teuthology.orchestra.run.trial038.stdout:Setting up ceph-volume (20.2.0-670-gadc19233-1jammy) ... 2026-02-18T00:42:22.026 INFO:teuthology.orchestra.run.trial038.stdout:Processing triggers for man-db (2.10.2-1) ... 2026-02-18T00:42:22.045 INFO:teuthology.orchestra.run.trial038.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.11) ... 2026-02-18T00:42:22.495 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:42:22.495 INFO:teuthology.orchestra.run.trial038.stdout:Running kernel seems to be up-to-date. 2026-02-18T00:42:22.496 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:42:22.496 INFO:teuthology.orchestra.run.trial038.stdout:The processor microcode seems to be up-to-date. 2026-02-18T00:42:22.496 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:42:22.496 INFO:teuthology.orchestra.run.trial038.stdout:No services need to be restarted. 2026-02-18T00:42:22.496 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:42:22.496 INFO:teuthology.orchestra.run.trial038.stdout:No containers need to be restarted. 2026-02-18T00:42:22.496 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:42:22.496 INFO:teuthology.orchestra.run.trial038.stdout:No user sessions are running outdated binaries. 2026-02-18T00:42:22.496 INFO:teuthology.orchestra.run.trial038.stdout: 2026-02-18T00:42:22.496 INFO:teuthology.orchestra.run.trial038.stdout:No VM guests are running outdated hypervisor (qemu) binaries on this host. 2026-02-18T00:42:23.136 INFO:teuthology.orchestra.run.trial038.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2026-02-18T00:42:23.141 DEBUG:teuthology.parallel:result is None 2026-02-18T00:42:23.142 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=adc1923395f2bcd6ae89b5d331d278364d1848ed 2026-02-18T00:42:23.275 DEBUG:teuthology.orchestra.run.trial038:> dpkg-query -W -f '${Version}' ceph 2026-02-18T00:42:23.285 INFO:teuthology.orchestra.run.trial038.stdout:20.2.0-670-gadc19233-1jammy 2026-02-18T00:42:23.286 INFO:teuthology.packaging:The installed version of ceph is 20.2.0-670-gadc19233-1jammy 2026-02-18T00:42:23.286 INFO:teuthology.task.install:The correct ceph version 20.2.0-670-gadc19233-1jammy is installed. 2026-02-18T00:42:23.287 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2026-02-18T00:42:23.288 DEBUG:teuthology.orchestra.run.trial038:> set -ex 2026-02-18T00:42:23.288 DEBUG:teuthology.orchestra.run.trial038:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2026-02-18T00:42:23.340 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2026-02-18T00:42:23.340 DEBUG:teuthology.orchestra.run.trial038:> set -ex 2026-02-18T00:42:23.340 DEBUG:teuthology.orchestra.run.trial038:> sudo dd of=/usr/bin/daemon-helper 2026-02-18T00:42:23.393 DEBUG:teuthology.orchestra.run.trial038:> sudo chmod a=rx -- /usr/bin/daemon-helper 2026-02-18T00:42:23.444 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2026-02-18T00:42:23.444 DEBUG:teuthology.orchestra.run.trial038:> set -ex 2026-02-18T00:42:23.445 DEBUG:teuthology.orchestra.run.trial038:> sudo dd of=/usr/bin/adjust-ulimits 2026-02-18T00:42:23.495 DEBUG:teuthology.orchestra.run.trial038:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2026-02-18T00:42:23.542 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2026-02-18T00:42:23.542 DEBUG:teuthology.orchestra.run.trial038:> set -ex 2026-02-18T00:42:23.542 DEBUG:teuthology.orchestra.run.trial038:> sudo dd of=/usr/bin/stdin-killer 2026-02-18T00:42:23.591 DEBUG:teuthology.orchestra.run.trial038:> sudo chmod a=rx -- /usr/bin/stdin-killer 2026-02-18T00:42:23.639 INFO:teuthology.run_tasks:Running task workunit... 2026-02-18T00:42:23.662 INFO:tasks.workunit:Pulling workunits from ref adc1923395f2bcd6ae89b5d331d278364d1848ed 2026-02-18T00:42:23.662 INFO:tasks.workunit:Making a separate scratch dir for every client... 2026-02-18T00:42:23.662 INFO:tasks.workunit:timeout=3h 2026-02-18T00:42:23.662 INFO:tasks.workunit:cleanup=True 2026-02-18T00:42:23.663 DEBUG:teuthology.orchestra.run.trial038:> stat -- /home/ubuntu/cephtest/mnt.0 2026-02-18T00:42:23.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-18T00:42:23.683 INFO:teuthology.orchestra.run.trial038.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.0': No such file or directory 2026-02-18T00:42:23.684 DEBUG:teuthology.orchestra.run.trial038:> mkdir -- /home/ubuntu/cephtest/mnt.0 2026-02-18T00:42:23.731 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.0 2026-02-18T00:42:23.731 DEBUG:teuthology.orchestra.run.trial038:> cd -- /home/ubuntu/cephtest/mnt.0 && mkdir -- client.0 2026-02-18T00:42:23.775 DEBUG:teuthology.orchestra.run.trial038:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://github.com/ceph/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout adc1923395f2bcd6ae89b5d331d278364d1848ed 2026-02-18T00:42:23.826 INFO:tasks.workunit.client.0.trial038.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2026-02-18T00:42:44.987 INFO:tasks.workunit.client.0.trial038.stderr:Note: switching to 'adc1923395f2bcd6ae89b5d331d278364d1848ed'. 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr: 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr:changes and commit them, and you can discard any commits you make in this 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr:state without impacting any branches by switching back to a branch. 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr: 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr:If you want to create a new branch to retain commits you create, you may 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr:do so (now or later) by using -c with the switch command. Example: 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr: 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr: git switch -c 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr: 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr:Or undo this operation with: 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr: 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr: git switch - 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr: 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2026-02-18T00:42:44.988 INFO:tasks.workunit.client.0.trial038.stderr: 2026-02-18T00:42:44.989 INFO:tasks.workunit.client.0.trial038.stderr:HEAD is now at adc1923395f Merge PR #67279 into tentacle 2026-02-18T00:42:44.997 DEBUG:teuthology.orchestra.run.trial038:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/standalone && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2026-02-18T00:42:45.045 DEBUG:teuthology.orchestra.run.trial038:> set -ex 2026-02-18T00:42:45.045 DEBUG:teuthology.orchestra.run.trial038:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2026-02-18T00:42:45.093 INFO:tasks.workunit:Running workunits matching mon-stretch on client.0... 2026-02-18T00:42:45.094 INFO:tasks.workunit:Running workunit mon-stretch/mon-stretch-fail-recovery.sh... 2026-02-18T00:42:45.094 DEBUG:teuthology.orchestra.run.trial038:workunit test mon-stretch/mon-stretch-fail-recovery.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=adc1923395f2bcd6ae89b5d331d278364d1848ed TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh 2026-02-18T00:42:45.146 INFO:tasks.workunit.client.0.trial038.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-18T00:42:45.151 INFO:tasks.workunit.client.0.trial038.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-18T00:42:45.152 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-18T00:42:45.152 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-18T00:42:45.152 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-18T00:42:45.152 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-18T00:42:45.152 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-18T00:42:45.152 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-18T00:42:45.152 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-18T00:42:45.152 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-18T00:42:45.152 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/mon-stretch-fail-recovery 2026-02-18T00:42:45.153 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:5: run: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:42:45.153 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:6: run: shift 2026-02-18T00:42:45.153 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:8: run: export CEPH_MON_A=127.0.0.1:7139 2026-02-18T00:42:45.153 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:8: run: CEPH_MON_A=127.0.0.1:7139 2026-02-18T00:42:45.153 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:9: run: export CEPH_MON_B=127.0.0.1:7141 2026-02-18T00:42:45.153 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:9: run: CEPH_MON_B=127.0.0.1:7141 2026-02-18T00:42:45.153 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:10: run: export CEPH_MON_C=127.0.0.1:7142 2026-02-18T00:42:45.153 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:10: run: CEPH_MON_C=127.0.0.1:7142 2026-02-18T00:42:45.153 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:11: run: export CEPH_MON_D=127.0.0.1:7143 2026-02-18T00:42:45.153 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:11: run: CEPH_MON_D=127.0.0.1:7143 2026-02-18T00:42:45.154 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:12: run: export CEPH_MON_E=127.0.0.1:7144 2026-02-18T00:42:45.154 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:12: run: CEPH_MON_E=127.0.0.1:7144 2026-02-18T00:42:45.154 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:13: run: export CEPH_ARGS 2026-02-18T00:42:45.154 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:14: run: uuidgen 2026-02-18T00:42:45.154 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:14: run: CEPH_ARGS+='--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none ' 2026-02-18T00:42:45.154 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:16: run: export 'BASE_CEPH_ARGS=--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none ' 2026-02-18T00:42:45.154 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:16: run: BASE_CEPH_ARGS='--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none ' 2026-02-18T00:42:45.154 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:17: run: CEPH_ARGS+=--mon-host=127.0.0.1:7139 2026-02-18T00:42:45.156 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:19: run: set 2026-02-18T00:42:45.156 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:19: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-18T00:42:45.156 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:19: run: local funcs=TEST_stretched_cluster_failover_add_three_osds 2026-02-18T00:42:45.157 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:20: run: for func in $funcs 2026-02-18T00:42:45.157 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:21: run: setup td/mon-stretch-fail-recovery 2026-02-18T00:42:45.157 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:42:45.157 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mon-stretch-fail-recovery 2026-02-18T00:42:45.157 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:42:45.157 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-18T00:42:45.157 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretch-fail-recovery KILL 2026-02-18T00:42:45.157 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:42:45.157 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:42:45.157 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:42:45.158 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:42:45.158 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:42:45.158 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:42:45.159 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-18T00:42:45.159 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-18T00:42:45.159 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-18T00:42:45.159 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-18T00:42:45.159 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-18T00:42:45.160 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-18T00:42:45.160 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:42:45.160 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-18T00:42:45.161 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-18T00:42:45.161 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:42:45.162 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-18T00:42:45.163 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-18T00:42:45.163 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretch-fail-recovery 2026-02-18T00:42:45.165 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-18T00:42:45.165 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:45.165 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:45.165 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.7520 2026-02-18T00:42:45.166 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-18T00:42:45.166 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-18T00:42:45.166 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mon-stretch-fail-recovery 2026-02-18T00:42:45.168 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-18T00:42:45.168 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:45.168 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:45.168 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.7520 2026-02-18T00:42:45.170 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-18T00:42:45.172 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-18T00:42:45.172 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-18T00:42:45.172 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-18T00:42:45.172 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mon-stretch-fail-recovery 1' TERM HUP INT 2026-02-18T00:42:45.172 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:22: run: TEST_stretched_cluster_failover_add_three_osds td/mon-stretch-fail-recovery 2026-02-18T00:42:45.172 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:27: TEST_stretched_cluster_failover_add_three_osds: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:42:45.172 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:28: TEST_stretched_cluster_failover_add_three_osds: local OSDS=8 2026-02-18T00:42:45.173 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:29: TEST_stretched_cluster_failover_add_three_osds: setup td/mon-stretch-fail-recovery 2026-02-18T00:42:45.173 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:42:45.173 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mon-stretch-fail-recovery 2026-02-18T00:42:45.173 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:42:45.173 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-18T00:42:45.173 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretch-fail-recovery KILL 2026-02-18T00:42:45.173 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:42:45.173 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:42:45.173 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:42:45.173 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:42:45.174 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:42:45.174 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:42:45.174 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-18T00:42:45.175 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-18T00:42:45.175 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-18T00:42:45.176 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-18T00:42:45.177 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-18T00:42:45.177 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-18T00:42:45.178 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:42:45.178 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-18T00:42:45.179 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-18T00:42:45.179 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:42:45.179 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-18T00:42:45.181 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-18T00:42:45.181 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretch-fail-recovery 2026-02-18T00:42:45.182 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-18T00:42:45.182 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:45.183 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:45.183 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.7520 2026-02-18T00:42:45.184 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-18T00:42:45.184 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-18T00:42:45.184 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mon-stretch-fail-recovery 2026-02-18T00:42:45.185 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-18T00:42:45.186 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:45.186 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:45.186 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.7520 2026-02-18T00:42:45.187 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-18T00:42:45.189 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-18T00:42:45.190 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-18T00:42:45.190 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mon-stretch-fail-recovery 1' TERM HUP INT 2026-02-18T00:42:45.190 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:31: TEST_stretched_cluster_failover_add_three_osds: run_mon td/mon-stretch-fail-recovery a --public-addr 127.0.0.1:7139 2026-02-18T00:42:45.190 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:42:45.190 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-18T00:42:45.190 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-18T00:42:45.190 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-18T00:42:45.191 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretch-fail-recovery/a 2026-02-18T00:42:45.191 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mon-stretch-fail-recovery/a --run-dir=td/mon-stretch-fail-recovery --public-addr 127.0.0.1:7139 2026-02-18T00:42:45.221 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-18T00:42:45.221 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:42:45.221 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:42:45.221 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:42:45.221 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:45.222 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:45.222 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:42:45.222 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretch-fail-recovery/a '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretch-fail-recovery/log --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7139 2026-02-18T00:42:45.250 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-18T00:42:45.251 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-18T00:42:45.251 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:42:45.251 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-18T00:42:45.251 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-18T00:42:45.252 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-18T00:42:45.253 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-18T00:42:45.253 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-18T00:42:45.253 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-18T00:42:45.253 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:42:45.253 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:45.253 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:45.254 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.7520/ceph-mon.a.asok 2026-02-18T00:42:45.254 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:42:45.254 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.7520/ceph-mon.a.asok config get fsid 2026-02-18T00:42:45.309 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-18T00:42:45.309 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:42:45.309 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-18T00:42:45.309 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-18T00:42:45.309 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-18T00:42:45.311 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-18T00:42:45.311 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-18T00:42:45.311 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-18T00:42:45.311 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:42:45.311 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:45.311 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:45.311 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.7520/ceph-mon.a.asok 2026-02-18T00:42:45.311 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:42:45.312 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.7520/ceph-mon.a.asok config get mon_host 2026-02-18T00:42:45.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:32: TEST_stretched_cluster_failover_add_three_osds: wait_for_quorum 300 1 2026-02-18T00:42:45.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2026-02-18T00:42:45.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=1 2026-02-18T00:42:45.366 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2026-02-18T00:42:45.366 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 1 ]] 2026-02-18T00:42:45.366 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-18T00:42:45.366 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-18T00:42:45.367 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771375665 2026-02-18T00:42:45.367 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-18T00:42:45.367 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771375365 -lt 1771375665 ]] 2026-02-18T00:42:45.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 1' 2026-02-18T00:42:45.368 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2026-02-18T00:42:45.480 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-18T00:42:45.480 INFO:tasks.workunit.client.0.trial038.stderr:{"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf","modified":"2026-02-18T00:42:45.201377Z","created":"2026-02-18T00:42:45.201377Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:42:45.481 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf","modified":"2026-02-18T00:42:45.201377Z","created":"2026-02-18T00:42:45.201377Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:42:45.481 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 1' 2026-02-18T00:42:45.492 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-18T00:42:45.492 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-18T00:42:45.492 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-18T00:42:45.492 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-18T00:42:45.493 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-18T00:42:45.493 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:34: TEST_stretched_cluster_failover_add_three_osds: run_mon td/mon-stretch-fail-recovery b --public-addr 127.0.0.1:7141 2026-02-18T00:42:45.493 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:42:45.493 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-18T00:42:45.493 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-18T00:42:45.493 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-18T00:42:45.493 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretch-fail-recovery/b 2026-02-18T00:42:45.493 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/mon-stretch-fail-recovery/b --run-dir=td/mon-stretch-fail-recovery --public-addr 127.0.0.1:7141 2026-02-18T00:42:45.511 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-18T00:42:45.512 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:42:45.512 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:42:45.512 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:42:45.512 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:45.512 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:45.512 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:42:45.513 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretch-fail-recovery/b '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretch-fail-recovery/log --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7141 2026-02-18T00:42:45.538 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-18T00:42:45.539 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-18T00:42:45.540 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:42:45.540 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-18T00:42:45.540 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-18T00:42:45.540 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-18T00:42:45.541 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-18T00:42:45.541 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-18T00:42:45.542 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-18T00:42:45.542 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:42:45.542 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:45.542 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:45.542 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.7520/ceph-mon.b.asok 2026-02-18T00:42:45.542 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:42:45.542 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.7520/ceph-mon.b.asok config get fsid 2026-02-18T00:42:45.597 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-18T00:42:45.597 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:42:45.597 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-18T00:42:45.597 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-18T00:42:45.598 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-18T00:42:45.598 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-18T00:42:45.598 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-18T00:42:45.598 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-18T00:42:45.598 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:42:45.598 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:45.598 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:45.598 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.7520/ceph-mon.b.asok 2026-02-18T00:42:45.599 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:42:45.599 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.7520/ceph-mon.b.asok config get mon_host 2026-02-18T00:42:45.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:35: TEST_stretched_cluster_failover_add_three_osds: CEPH_ARGS='--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141' 2026-02-18T00:42:45.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:36: TEST_stretched_cluster_failover_add_three_osds: wait_for_quorum 300 2 2026-02-18T00:42:45.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2026-02-18T00:42:45.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=2 2026-02-18T00:42:45.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2026-02-18T00:42:45.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 2 ]] 2026-02-18T00:42:45.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-18T00:42:45.653 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-18T00:42:45.654 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771375665 2026-02-18T00:42:45.654 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-18T00:42:45.655 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771375365 -lt 1771375665 ]] 2026-02-18T00:42:45.655 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-18T00:42:45.656 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2026-02-18T00:42:51.771 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-18T00:42:51.771 INFO:tasks.workunit.client.0.trial038.stderr:{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":2,"fsid":"e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf","modified":"2026-02-18T00:42:45.537663Z","created":"2026-02-18T00:42:45.201377Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:42:51.772 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":2,"fsid":"e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf","modified":"2026-02-18T00:42:45.537663Z","created":"2026-02-18T00:42:45.201377Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:42:51.772 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-18T00:42:51.781 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-18T00:42:51.781 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-18T00:42:51.781 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-18T00:42:51.782 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-18T00:42:51.782 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-18T00:42:51.782 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:38: TEST_stretched_cluster_failover_add_three_osds: run_mon td/mon-stretch-fail-recovery c --public-addr 127.0.0.1:7142 2026-02-18T00:42:51.782 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:42:51.782 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-18T00:42:51.782 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-18T00:42:51.782 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-18T00:42:51.782 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretch-fail-recovery/c 2026-02-18T00:42:51.782 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/mon-stretch-fail-recovery/c --run-dir=td/mon-stretch-fail-recovery --public-addr 127.0.0.1:7142 2026-02-18T00:42:51.801 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-18T00:42:51.802 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:42:51.802 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:42:51.802 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:42:51.802 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:51.802 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:51.802 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:42:51.803 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretch-fail-recovery/c '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretch-fail-recovery/log --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7142 2026-02-18T00:42:51.831 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-18T00:42:51.833 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-18T00:42:51.833 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:42:51.833 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-18T00:42:51.834 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-18T00:42:51.834 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-18T00:42:51.834 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-18T00:42:51.834 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-18T00:42:51.834 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-18T00:42:51.834 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:42:51.834 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:51.834 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:51.834 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.7520/ceph-mon.c.asok 2026-02-18T00:42:51.835 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:42:51.835 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.7520/ceph-mon.c.asok config get fsid 2026-02-18T00:42:51.898 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-18T00:42:51.898 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:42:51.899 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-18T00:42:51.899 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-18T00:42:51.899 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-18T00:42:51.900 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-18T00:42:51.901 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-18T00:42:51.901 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-18T00:42:51.901 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:42:51.901 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:51.901 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:51.901 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.7520/ceph-mon.c.asok 2026-02-18T00:42:51.901 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:42:51.901 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.7520/ceph-mon.c.asok config get mon_host 2026-02-18T00:42:51.954 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:39: TEST_stretched_cluster_failover_add_three_osds: CEPH_ARGS='--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142' 2026-02-18T00:42:51.955 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:40: TEST_stretched_cluster_failover_add_three_osds: wait_for_quorum 300 3 2026-02-18T00:42:51.955 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2026-02-18T00:42:51.955 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-18T00:42:51.955 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2026-02-18T00:42:51.955 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-18T00:42:51.955 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-18T00:42:51.955 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-18T00:42:51.956 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771375671 2026-02-18T00:42:51.956 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-18T00:42:51.957 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771375371 -lt 1771375671 ]] 2026-02-18T00:42:51.957 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-18T00:42:51.957 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2026-02-18T00:42:58.071 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-18T00:42:58.071 INFO:tasks.workunit.client.0.trial038.stderr:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":3,"fsid":"e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf","modified":"2026-02-18T00:42:51.833182Z","created":"2026-02-18T00:42:45.201377Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:42:58.072 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":3,"fsid":"e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf","modified":"2026-02-18T00:42:51.833182Z","created":"2026-02-18T00:42:45.201377Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:42:58.072 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-18T00:42:58.084 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-18T00:42:58.084 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-18T00:42:58.084 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-18T00:42:58.084 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-18T00:42:58.084 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-18T00:42:58.084 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:42: TEST_stretched_cluster_failover_add_three_osds: run_mon td/mon-stretch-fail-recovery d --public-addr 127.0.0.1:7143 2026-02-18T00:42:58.084 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:42:58.085 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-18T00:42:58.085 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2026-02-18T00:42:58.085 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-18T00:42:58.085 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretch-fail-recovery/d 2026-02-18T00:42:58.085 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/mon-stretch-fail-recovery/d --run-dir=td/mon-stretch-fail-recovery --public-addr 127.0.0.1:7143 2026-02-18T00:42:58.103 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-18T00:42:58.103 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:42:58.104 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:42:58.104 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:42:58.104 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:58.104 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:58.104 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:42:58.105 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretch-fail-recovery/d '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretch-fail-recovery/log --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7143 2026-02-18T00:42:58.131 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-18T00:42:58.132 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2026-02-18T00:42:58.132 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:42:58.132 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-18T00:42:58.132 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-18T00:42:58.132 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-18T00:42:58.132 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-18T00:42:58.133 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-18T00:42:58.133 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-18T00:42:58.134 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:42:58.134 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:58.134 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:58.134 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.7520/ceph-mon.d.asok 2026-02-18T00:42:58.134 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:42:58.134 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.7520/ceph-mon.d.asok config get fsid 2026-02-18T00:42:58.197 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2026-02-18T00:42:58.197 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:42:58.198 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-18T00:42:58.198 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-18T00:42:58.198 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-18T00:42:58.199 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-18T00:42:58.199 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-18T00:42:58.199 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-18T00:42:58.199 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:42:58.199 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:42:58.200 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:42:58.200 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.7520/ceph-mon.d.asok 2026-02-18T00:42:58.200 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:42:58.200 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.7520/ceph-mon.d.asok config get mon_host 2026-02-18T00:42:58.252 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:43: TEST_stretched_cluster_failover_add_three_osds: CEPH_ARGS='--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143' 2026-02-18T00:42:58.253 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:44: TEST_stretched_cluster_failover_add_three_osds: wait_for_quorum 300 4 2026-02-18T00:42:58.253 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2026-02-18T00:42:58.253 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2026-02-18T00:42:58.253 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2026-02-18T00:42:58.253 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2026-02-18T00:42:58.253 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-18T00:42:58.253 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-18T00:42:58.254 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771375678 2026-02-18T00:42:58.254 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-18T00:42:58.254 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771375378 -lt 1771375678 ]] 2026-02-18T00:42:58.254 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-18T00:42:58.255 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2026-02-18T00:43:05.782 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-18T00:43:05.782 INFO:tasks.workunit.client.0.trial038.stderr:{"election_epoch":16,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":4,"fsid":"e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf","modified":"2026-02-18T00:42:58.132923Z","created":"2026-02-18T00:42:45.201377Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:43:05.783 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":16,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":4,"fsid":"e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf","modified":"2026-02-18T00:42:58.132923Z","created":"2026-02-18T00:42:45.201377Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:43:05.783 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-18T00:43:05.792 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-18T00:43:05.792 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-18T00:43:05.792 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-18T00:43:05.792 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-18T00:43:05.792 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-18T00:43:05.792 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:46: TEST_stretched_cluster_failover_add_three_osds: run_mon td/mon-stretch-fail-recovery e --public-addr 127.0.0.1:7144 2026-02-18T00:43:05.792 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:43:05.792 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-18T00:43:05.793 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=e 2026-02-18T00:43:05.793 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-18T00:43:05.793 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretch-fail-recovery/e 2026-02-18T00:43:05.793 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id e --mkfs --mon-data=td/mon-stretch-fail-recovery/e --run-dir=td/mon-stretch-fail-recovery --public-addr 127.0.0.1:7144 2026-02-18T00:43:05.814 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-18T00:43:05.814 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:43:05.814 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:43:05.814 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:43:05.814 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:05.814 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:05.815 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:05.815 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretch-fail-recovery/e '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretch-fail-recovery/log --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7144 2026-02-18T00:43:05.843 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-18T00:43:05.844 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e fsid 2026-02-18T00:43:05.844 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:43:05.845 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2026-02-18T00:43:05.845 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-18T00:43:05.846 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-18T00:43:05.846 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2026-02-18T00:43:05.846 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2026-02-18T00:43:05.846 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2026-02-18T00:43:05.846 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:43:05.846 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:05.847 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:05.847 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.7520/ceph-mon.e.asok 2026-02-18T00:43:05.847 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:43:05.847 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.7520/ceph-mon.e.asok config get fsid 2026-02-18T00:43:05.901 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e mon_host 2026-02-18T00:43:05.901 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:43:05.902 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2026-02-18T00:43:05.902 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-18T00:43:05.902 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-18T00:43:05.902 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2026-02-18T00:43:05.902 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2026-02-18T00:43:05.902 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2026-02-18T00:43:05.903 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:43:05.903 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:05.903 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:05.903 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.7520/ceph-mon.e.asok 2026-02-18T00:43:05.903 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:43:05.904 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.7520/ceph-mon.e.asok config get mon_host 2026-02-18T00:43:05.955 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:47: TEST_stretched_cluster_failover_add_three_osds: CEPH_ARGS='--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:43:05.956 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:48: TEST_stretched_cluster_failover_add_three_osds: wait_for_quorum 300 5 2026-02-18T00:43:05.956 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2026-02-18T00:43:05.956 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=5 2026-02-18T00:43:05.956 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2026-02-18T00:43:05.956 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 5 ]] 2026-02-18T00:43:05.956 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-18T00:43:05.956 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-18T00:43:05.957 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771375685 2026-02-18T00:43:05.957 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-18T00:43:05.958 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771375385 -lt 1771375685 ]] 2026-02-18T00:43:05.958 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 5' 2026-02-18T00:43:05.958 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2026-02-18T00:43:12.069 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-18T00:43:12.069 INFO:tasks.workunit.client.0.trial038.stderr:{"election_epoch":22,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":5,"fsid":"e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf","modified":"2026-02-18T00:43:05.846176Z","created":"2026-02-18T00:42:45.201377Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7144","nonce":0}]},"addr":"127.0.0.1:7144/0","public_addr":"127.0.0.1:7144/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:43:12.069 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 5' 2026-02-18T00:43:12.070 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":22,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":5,"fsid":"e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf","modified":"2026-02-18T00:43:05.846176Z","created":"2026-02-18T00:42:45.201377Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7144","nonce":0}]},"addr":"127.0.0.1:7144/0","public_addr":"127.0.0.1:7144/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:43:12.079 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-18T00:43:12.079 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-18T00:43:12.079 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-18T00:43:12.079 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-18T00:43:12.079 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-18T00:43:12.079 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:50: TEST_stretched_cluster_failover_add_three_osds: ceph mon set election_strategy connectivity 2026-02-18T00:43:12.200 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:51: TEST_stretched_cluster_failover_add_three_osds: ceph mon add disallowed_leader e 2026-02-18T00:43:12.337 INFO:tasks.workunit.client.0.trial038.stderr:mon.e is already disallowed 2026-02-18T00:43:12.346 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:53: TEST_stretched_cluster_failover_add_three_osds: run_mgr td/mon-stretch-fail-recovery x 2026-02-18T00:43:12.346 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:43:12.346 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-18T00:43:12.346 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-18T00:43:12.346 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-18T00:43:12.346 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mon-stretch-fail-recovery/x 2026-02-18T00:43:12.346 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-18T00:43:12.466 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-18T00:43:12.467 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:43:12.467 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:43:12.467 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:43:12.467 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:12.467 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:12.467 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:12.467 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-18T00:43:12.468 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mon-stretch-fail-recovery/x '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-18T00:43:12.488 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:54: TEST_stretched_cluster_failover_add_three_osds: run_mgr td/mon-stretch-fail-recovery y 2026-02-18T00:43:12.489 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:43:12.489 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-18T00:43:12.489 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=y 2026-02-18T00:43:12.489 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-18T00:43:12.489 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mon-stretch-fail-recovery/y 2026-02-18T00:43:12.489 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-18T00:43:12.607 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-18T00:43:12.607 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:43:12.607 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:43:12.608 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:43:12.608 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:12.608 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:12.608 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:12.608 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-18T00:43:12.609 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id y --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mon-stretch-fail-recovery/y '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-18T00:43:15.629 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:55: TEST_stretched_cluster_failover_add_three_osds: run_mgr td/mon-stretch-fail-recovery z 2026-02-18T00:43:15.629 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:43:15.629 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-18T00:43:15.629 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=z 2026-02-18T00:43:15.629 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-18T00:43:15.629 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mon-stretch-fail-recovery/z 2026-02-18T00:43:15.629 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-18T00:43:15.848 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-18T00:43:15.849 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:43:15.849 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:43:15.849 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:43:15.849 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:15.849 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:15.849 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:15.850 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-18T00:43:15.850 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id z --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mon-stretch-fail-recovery/z '--log-file=td/mon-stretch-fail-recovery/$name.log' '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --run-dir=td/mon-stretch-fail-recovery '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-18T00:43:15.866 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: expr 8 - 1 2026-02-18T00:43:15.867 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: seq 0 7 2026-02-18T00:43:15.872 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-18T00:43:15.872 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 0 2026-02-18T00:43:15.872 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:43:15.872 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:43:15.872 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-18T00:43:15.872 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:43:15.872 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/0 2026-02-18T00:43:15.873 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:43:15.873 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:43:15.873 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:43:15.873 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:43:15.873 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/0' 2026-02-18T00:43:15.873 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/0/journal' 2026-02-18T00:43:15.873 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:43:15.873 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:43:15.873 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2026-02-18T00:43:15.873 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:43:15.873 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:43:15.874 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:43:15.874 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:43:15.874 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:15.874 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:15.874 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:15.874 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:15.874 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:43:15.874 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:43:15.874 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:43:15.874 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2026-02-18T00:43:15.874 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2026-02-18T00:43:15.874 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:43:15.874 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:43:15.875 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:43:15.875 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:43:15.875 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:43:15.875 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:43:15.875 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/0 2026-02-18T00:43:15.875 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:43:15.876 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=afbd8ad2-6531-4d12-8e7c-899bff7e5370 2026-02-18T00:43:15.876 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 afbd8ad2-6531-4d12-8e7c-899bff7e5370' 2026-02-18T00:43:15.876 INFO:tasks.workunit.client.0.trial038.stdout:add osd0 afbd8ad2-6531-4d12-8e7c-899bff7e5370 2026-02-18T00:43:15.876 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:43:15.883 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAjC5Vp7++RNBAALMoljVQYEbrLQxjZh0g9gA== 2026-02-18T00:43:15.884 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAjC5Vp7++RNBAALMoljVQYEbrLQxjZh0g9gA=="}' 2026-02-18T00:43:15.884 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new afbd8ad2-6531-4d12-8e7c-899bff7e5370 -i td/mon-stretch-fail-recovery/0/new.json 2026-02-18T00:43:16.104 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:43:16.116 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/0/new.json 2026-02-18T00:43:16.117 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/0 --osd-journal=td/mon-stretch-fail-recovery/0/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAjC5Vp7++RNBAALMoljVQYEbrLQxjZh0g9gA== --osd-uuid afbd8ad2-6531-4d12-8e7c-899bff7e5370 2026-02-18T00:43:16.142 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:16.135+0000 7f2bdf1b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:16.144 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:16.139+0000 7f2bdf1b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:16.145 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:16.139+0000 7f2bdf1b0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:16.146 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:16.139+0000 7f2bdf1b0a40 -1 bdev(0x557dd06e6800 td/mon-stretch-fail-recovery/0/block) open stat got: (1) Operation not permitted 2026-02-18T00:43:16.146 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:16.139+0000 7f2bdf1b0a40 -1 bluestore(td/mon-stretch-fail-recovery/0) _read_fsid unparsable uuid 2026-02-18T00:43:17.068 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/0/keyring 2026-02-18T00:43:17.069 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:43:17.070 INFO:tasks.workunit.client.0.trial038.stdout:adding osd0 key to auth repository 2026-02-18T00:43:17.070 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-18T00:43:17.070 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:43:17.367 INFO:tasks.workunit.client.0.trial038.stdout:start osd.0 2026-02-18T00:43:17.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-18T00:43:17.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/0 --osd-journal=td/mon-stretch-fail-recovery/0/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:43:17.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:43:17.369 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:43:17.369 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:43:17.392 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:17.387+0000 7f4684f01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:17.394 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:17.387+0000 7f4684f01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:17.395 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:17.387+0000 7f4684f01a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:17.588 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-18T00:43:17.588 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:43:17.588 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-18T00:43:17.588 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:43:17.589 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:43:17.589 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:17.589 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:43:17.589 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:43:17.590 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:17.590 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-18T00:43:17.845 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:43:17.957 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:17.951+0000 7f4684f01a40 -1 Falling back to public interface 2026-02-18T00:43:18.744 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:18.739+0000 7f4684f01a40 -1 osd.0 0 log_to_monitors true 2026-02-18T00:43:18.846 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:43:18.846 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:18.847 INFO:tasks.workunit.client.0.trial038.stdout:1 2026-02-18T00:43:18.847 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-18T00:43:18.848 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:18.848 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-18T00:43:19.057 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:43:20.059 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:43:20.059 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:20.060 INFO:tasks.workunit.client.0.trial038.stdout:2 2026-02-18T00:43:20.060 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-18T00:43:20.061 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:20.061 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-18T00:43:20.275 INFO:tasks.workunit.client.0.trial038.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/1727726895,v1:127.0.0.1:6803/1727726895] [v2:127.0.0.1:6804/1727726895,v1:127.0.0.1:6805/1727726895] exists,up afbd8ad2-6531-4d12-8e7c-899bff7e5370 2026-02-18T00:43:20.279 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:43:20.279 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:43:20.280 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:43:20.280 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-18T00:43:20.280 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 1 2026-02-18T00:43:20.280 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:43:20.280 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:43:20.280 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-18T00:43:20.280 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:43:20.280 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/1 2026-02-18T00:43:20.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:43:20.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:43:20.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:43:20.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:43:20.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/1' 2026-02-18T00:43:20.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/1/journal' 2026-02-18T00:43:20.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:43:20.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:43:20.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2026-02-18T00:43:20.282 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:43:20.282 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:43:20.282 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:43:20.282 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:43:20.282 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:20.282 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:20.282 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:20.282 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:20.282 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:43:20.282 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:43:20.282 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:43:20.282 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2026-02-18T00:43:20.282 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2026-02-18T00:43:20.283 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:43:20.283 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:43:20.283 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:43:20.283 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:43:20.283 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:43:20.283 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:43:20.283 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/1 2026-02-18T00:43:20.283 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:43:20.283 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=befd336d-572d-490c-a35a-7435bfce238c 2026-02-18T00:43:20.284 INFO:tasks.workunit.client.0.trial038.stdout:add osd1 befd336d-572d-490c-a35a-7435bfce238c 2026-02-18T00:43:20.284 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 befd336d-572d-490c-a35a-7435bfce238c' 2026-02-18T00:43:20.285 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:43:20.290 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAoC5VpCp0wERAAYAGnuuwtPHLxu+DLqgf2rA== 2026-02-18T00:43:20.290 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAoC5VpCp0wERAAYAGnuuwtPHLxu+DLqgf2rA=="}' 2026-02-18T00:43:20.290 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new befd336d-572d-490c-a35a-7435bfce238c -i td/mon-stretch-fail-recovery/1/new.json 2026-02-18T00:43:20.509 INFO:tasks.workunit.client.0.trial038.stdout:1 2026-02-18T00:43:20.520 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/1/new.json 2026-02-18T00:43:20.521 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/1 --osd-journal=td/mon-stretch-fail-recovery/1/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAoC5VpCp0wERAAYAGnuuwtPHLxu+DLqgf2rA== --osd-uuid befd336d-572d-490c-a35a-7435bfce238c 2026-02-18T00:43:20.545 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:20.539+0000 7f74b8e0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:20.546 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:20.539+0000 7f74b8e0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:20.547 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:20.543+0000 7f74b8e0ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:20.548 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:20.543+0000 7f74b8e0ea40 -1 bdev(0x555aa613a800 td/mon-stretch-fail-recovery/1/block) open stat got: (1) Operation not permitted 2026-02-18T00:43:20.548 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:20.543+0000 7f74b8e0ea40 -1 bluestore(td/mon-stretch-fail-recovery/1) _read_fsid unparsable uuid 2026-02-18T00:43:21.215 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/1/keyring 2026-02-18T00:43:21.215 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:43:21.216 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-18T00:43:21.216 INFO:tasks.workunit.client.0.trial038.stdout:adding osd1 key to auth repository 2026-02-18T00:43:21.217 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:43:21.506 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-18T00:43:21.506 INFO:tasks.workunit.client.0.trial038.stdout:start osd.1 2026-02-18T00:43:21.507 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/1 --osd-journal=td/mon-stretch-fail-recovery/1/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:43:21.507 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:43:21.508 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:43:21.508 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:43:21.528 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:21.523+0000 7f1066563a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:21.530 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:21.523+0000 7f1066563a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:21.531 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:21.523+0000 7f1066563a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:21.722 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-18T00:43:21.722 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:43:21.722 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-18T00:43:21.723 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:43:21.723 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:43:21.723 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:21.723 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:43:21.724 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:43:21.724 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:21.724 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-18T00:43:21.938 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:43:22.025 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:22.019+0000 7f1066563a40 -1 Falling back to public interface 2026-02-18T00:43:22.637 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:22.631+0000 7f1066563a40 -1 osd.1 0 log_to_monitors true 2026-02-18T00:43:22.939 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:43:22.940 INFO:tasks.workunit.client.0.trial038.stdout:1 2026-02-18T00:43:22.940 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:22.940 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-18T00:43:22.940 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:22.941 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-18T00:43:23.153 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:43:24.154 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:43:24.155 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:24.155 INFO:tasks.workunit.client.0.trial038.stdout:2 2026-02-18T00:43:24.156 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-18T00:43:24.156 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:24.156 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-18T00:43:24.361 INFO:tasks.workunit.client.0.trial038.stdout:osd.1 up in weight 1 up_from 9 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2581527721,v1:127.0.0.1:6811/2581527721] [v2:127.0.0.1:6812/2581527721,v1:127.0.0.1:6813/2581527721] exists,up befd336d-572d-490c-a35a-7435bfce238c 2026-02-18T00:43:24.364 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:43:24.364 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:43:24.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:43:24.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-18T00:43:24.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 2 2026-02-18T00:43:24.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:43:24.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:43:24.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-18T00:43:24.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:43:24.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/2 2026-02-18T00:43:24.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:43:24.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:43:24.366 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:43:24.366 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:43:24.366 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/2' 2026-02-18T00:43:24.366 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/2/journal' 2026-02-18T00:43:24.366 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:43:24.366 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:43:24.366 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2026-02-18T00:43:24.366 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:43:24.366 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:43:24.366 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:43:24.367 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:43:24.367 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:24.367 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:24.367 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:24.367 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:24.367 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:43:24.367 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:43:24.367 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:43:24.367 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2026-02-18T00:43:24.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2026-02-18T00:43:24.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:43:24.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:43:24.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:43:24.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:43:24.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:43:24.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:43:24.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/2 2026-02-18T00:43:24.369 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:43:24.370 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=58252d35-5e00-4ffa-a9fc-84fccef8873e 2026-02-18T00:43:24.370 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 58252d35-5e00-4ffa-a9fc-84fccef8873e' 2026-02-18T00:43:24.370 INFO:tasks.workunit.client.0.trial038.stdout:add osd2 58252d35-5e00-4ffa-a9fc-84fccef8873e 2026-02-18T00:43:24.370 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:43:24.378 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAsC5Vpd7BsFhAANR75lediAVg7c6y4b2akgg== 2026-02-18T00:43:24.378 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAsC5Vpd7BsFhAANR75lediAVg7c6y4b2akgg=="}' 2026-02-18T00:43:24.378 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 58252d35-5e00-4ffa-a9fc-84fccef8873e -i td/mon-stretch-fail-recovery/2/new.json 2026-02-18T00:43:24.597 INFO:tasks.workunit.client.0.trial038.stdout:2 2026-02-18T00:43:24.608 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/2/new.json 2026-02-18T00:43:24.609 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/2 --osd-journal=td/mon-stretch-fail-recovery/2/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAsC5Vpd7BsFhAANR75lediAVg7c6y4b2akgg== --osd-uuid 58252d35-5e00-4ffa-a9fc-84fccef8873e 2026-02-18T00:43:24.632 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:24.627+0000 7faa84d04a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:24.633 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:24.627+0000 7faa84d04a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:24.635 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:24.627+0000 7faa84d04a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:24.636 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:24.631+0000 7faa84d04a40 -1 bdev(0x557247084800 td/mon-stretch-fail-recovery/2/block) open stat got: (1) Operation not permitted 2026-02-18T00:43:24.636 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:24.631+0000 7faa84d04a40 -1 bluestore(td/mon-stretch-fail-recovery/2) _read_fsid unparsable uuid 2026-02-18T00:43:25.311 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/2/keyring 2026-02-18T00:43:25.311 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:43:25.312 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-18T00:43:25.312 INFO:tasks.workunit.client.0.trial038.stdout:adding osd2 key to auth repository 2026-02-18T00:43:25.313 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:43:25.599 INFO:tasks.workunit.client.0.trial038.stdout:start osd.2 2026-02-18T00:43:25.601 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-18T00:43:25.601 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/2 --osd-journal=td/mon-stretch-fail-recovery/2/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:43:25.601 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:43:25.601 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:43:25.601 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:43:25.622 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:25.615+0000 7f03b7ab0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:25.624 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:25.619+0000 7f03b7ab0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:25.625 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:25.619+0000 7f03b7ab0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:25.818 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-18T00:43:25.818 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:43:25.819 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-18T00:43:25.819 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:43:25.819 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:43:25.819 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:25.819 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:43:25.819 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:43:25.820 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:25.820 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-18T00:43:26.029 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:43:26.041 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:26.035+0000 7f03b7ab0a40 -1 Falling back to public interface 2026-02-18T00:43:26.746 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:26.739+0000 7f03b7ab0a40 -1 osd.2 0 log_to_monitors true 2026-02-18T00:43:27.030 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:43:27.031 INFO:tasks.workunit.client.0.trial038.stdout:1 2026-02-18T00:43:27.031 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:27.032 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-18T00:43:27.032 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:27.032 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-18T00:43:27.249 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:43:28.251 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:43:28.251 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:28.251 INFO:tasks.workunit.client.0.trial038.stdout:2 2026-02-18T00:43:28.252 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-18T00:43:28.252 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:28.252 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-18T00:43:31.460 INFO:tasks.workunit.client.0.trial038.stdout:osd.2 up in weight 1 up_from 13 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/1294821831,v1:127.0.0.1:6819/1294821831] [v2:127.0.0.1:6820/1294821831,v1:127.0.0.1:6821/1294821831] exists,up 58252d35-5e00-4ffa-a9fc-84fccef8873e 2026-02-18T00:43:31.465 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:43:31.466 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:43:31.466 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:43:31.466 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-18T00:43:31.466 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 3 2026-02-18T00:43:31.466 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:43:31.466 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:43:31.466 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-18T00:43:31.466 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:43:31.467 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/3 2026-02-18T00:43:31.467 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:43:31.467 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:43:31.467 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:43:31.467 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:43:31.467 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/3' 2026-02-18T00:43:31.467 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/3/journal' 2026-02-18T00:43:31.467 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:43:31.467 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:43:31.467 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2026-02-18T00:43:31.467 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:43:31.467 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:43:31.468 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:43:31.468 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:43:31.468 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:31.468 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:31.468 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:31.468 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:31.468 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:43:31.468 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:43:31.468 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:43:31.468 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2026-02-18T00:43:31.468 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2026-02-18T00:43:31.468 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:43:31.468 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:43:31.469 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:43:31.469 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:43:31.469 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:43:31.469 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:43:31.469 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/3 2026-02-18T00:43:31.469 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:43:31.469 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=69709b12-fea7-4085-a264-7cba14931aca 2026-02-18T00:43:31.469 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 69709b12-fea7-4085-a264-7cba14931aca' 2026-02-18T00:43:31.469 INFO:tasks.workunit.client.0.trial038.stdout:add osd3 69709b12-fea7-4085-a264-7cba14931aca 2026-02-18T00:43:31.470 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:43:31.477 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQAzC5VpxCVWHBAABnkgzMbO877QMdZeHUdOCg== 2026-02-18T00:43:31.477 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQAzC5VpxCVWHBAABnkgzMbO877QMdZeHUdOCg=="}' 2026-02-18T00:43:31.477 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 69709b12-fea7-4085-a264-7cba14931aca -i td/mon-stretch-fail-recovery/3/new.json 2026-02-18T00:43:31.697 INFO:tasks.workunit.client.0.trial038.stdout:3 2026-02-18T00:43:31.707 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/3/new.json 2026-02-18T00:43:31.707 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/3 --osd-journal=td/mon-stretch-fail-recovery/3/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQAzC5VpxCVWHBAABnkgzMbO877QMdZeHUdOCg== --osd-uuid 69709b12-fea7-4085-a264-7cba14931aca 2026-02-18T00:43:31.723 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:31.715+0000 7f43aba05a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:31.725 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:31.719+0000 7f43aba05a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:31.726 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:31.719+0000 7f43aba05a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:31.726 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:31.719+0000 7f43aba05a40 -1 bdev(0x561755d56800 td/mon-stretch-fail-recovery/3/block) open stat got: (1) Operation not permitted 2026-02-18T00:43:31.726 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:31.719+0000 7f43aba05a40 -1 bluestore(td/mon-stretch-fail-recovery/3) _read_fsid unparsable uuid 2026-02-18T00:43:32.402 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/3/keyring 2026-02-18T00:43:32.402 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:43:32.403 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-18T00:43:32.403 INFO:tasks.workunit.client.0.trial038.stdout:adding osd3 key to auth repository 2026-02-18T00:43:32.404 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:43:32.700 INFO:tasks.workunit.client.0.trial038.stdout:start osd.3 2026-02-18T00:43:32.701 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-18T00:43:32.701 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/3 --osd-journal=td/mon-stretch-fail-recovery/3/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:43:32.701 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:43:32.701 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:43:32.701 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:43:32.716 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:32.711+0000 7fe713563a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:32.717 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:32.711+0000 7fe713563a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:32.718 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:32.711+0000 7fe713563a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:32.918 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-18T00:43:32.918 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:43:32.918 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-18T00:43:32.918 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:43:32.918 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:43:32.918 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:32.918 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:43:32.918 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:43:32.918 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:32.918 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-18T00:43:33.129 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:43:33.141 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:33.135+0000 7fe713563a40 -1 Falling back to public interface 2026-02-18T00:43:33.849 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:33.843+0000 7fe713563a40 -1 osd.3 0 log_to_monitors true 2026-02-18T00:43:34.130 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:43:34.130 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:34.130 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-18T00:43:34.131 INFO:tasks.workunit.client.0.trial038.stdout:1 2026-02-18T00:43:34.131 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:34.131 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-18T00:43:34.346 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:43:35.348 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:43:35.348 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:35.349 INFO:tasks.workunit.client.0.trial038.stdout:2 2026-02-18T00:43:35.349 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-18T00:43:35.349 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:35.349 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-18T00:43:35.558 INFO:tasks.workunit.client.0.trial038.stdout:osd.3 up in weight 1 up_from 17 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/2636759975,v1:127.0.0.1:6827/2636759975] [v2:127.0.0.1:6828/2636759975,v1:127.0.0.1:6829/2636759975] exists,up 69709b12-fea7-4085-a264-7cba14931aca 2026-02-18T00:43:35.560 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:43:35.560 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:43:35.560 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:43:35.560 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-18T00:43:35.560 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 4 2026-02-18T00:43:35.560 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:43:35.560 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:43:35.560 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=4 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/4 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/4' 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/4/journal' 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:43:35.561 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:35.562 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:35.562 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/4 2026-02-18T00:43:35.563 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:43:35.564 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=ddf62834-f7d1-48cf-a44f-d1907112ade4 2026-02-18T00:43:35.564 INFO:tasks.workunit.client.0.trial038.stdout:add osd4 ddf62834-f7d1-48cf-a44f-d1907112ade4 2026-02-18T00:43:35.564 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd4 ddf62834-f7d1-48cf-a44f-d1907112ade4' 2026-02-18T00:43:35.564 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:43:35.576 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA3C5VpX3dFIhAAsMwAU7J/VLzy2WN5LV7bYQ== 2026-02-18T00:43:35.577 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA3C5VpX3dFIhAAsMwAU7J/VLzy2WN5LV7bYQ=="}' 2026-02-18T00:43:35.577 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new ddf62834-f7d1-48cf-a44f-d1907112ade4 -i td/mon-stretch-fail-recovery/4/new.json 2026-02-18T00:43:35.792 INFO:tasks.workunit.client.0.trial038.stdout:4 2026-02-18T00:43:35.801 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/4/new.json 2026-02-18T00:43:35.802 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 4 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/4 --osd-journal=td/mon-stretch-fail-recovery/4/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA3C5VpX3dFIhAAsMwAU7J/VLzy2WN5LV7bYQ== --osd-uuid ddf62834-f7d1-48cf-a44f-d1907112ade4 2026-02-18T00:43:35.817 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:35.811+0000 7ff4ce1eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:35.819 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:35.811+0000 7ff4ce1eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:35.820 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:35.815+0000 7ff4ce1eca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:35.820 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:35.815+0000 7ff4ce1eca40 -1 bdev(0x555a06b8c800 td/mon-stretch-fail-recovery/4/block) open stat got: (1) Operation not permitted 2026-02-18T00:43:35.820 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:35.815+0000 7ff4ce1eca40 -1 bluestore(td/mon-stretch-fail-recovery/4) _read_fsid unparsable uuid 2026-02-18T00:43:36.542 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/4/keyring 2026-02-18T00:43:36.542 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:43:36.543 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd4 key to auth repository 2026-02-18T00:43:36.543 INFO:tasks.workunit.client.0.trial038.stdout:adding osd4 key to auth repository 2026-02-18T00:43:36.544 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/4/keyring auth add osd.4 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:43:36.833 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.4 2026-02-18T00:43:36.834 INFO:tasks.workunit.client.0.trial038.stdout:start osd.4 2026-02-18T00:43:36.835 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 4 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/4 --osd-journal=td/mon-stretch-fail-recovery/4/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:43:36.835 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:43:36.835 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:43:36.835 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:43:36.848 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:36.843+0000 7f52d42c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:36.850 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:36.843+0000 7f52d42c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:36.851 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:36.843+0000 7f52d42c9a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:37.047 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 4 2026-02-18T00:43:37.047 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:43:37.048 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=4 2026-02-18T00:43:37.048 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:43:37.048 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:43:37.048 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:37.048 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:43:37.048 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:43:37.049 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:37.049 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.4 up' 2026-02-18T00:43:37.253 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:37.247+0000 7f52d42c9a40 -1 Falling back to public interface 2026-02-18T00:43:37.817 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:37.811+0000 7f52d42c9a40 -1 osd.4 0 log_to_monitors true 2026-02-18T00:43:40.260 INFO:tasks.workunit.client.0.trial038.stdout:osd.4 up in weight 1 up_from 21 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/2132771434,v1:127.0.0.1:6835/2132771434] [v2:127.0.0.1:6836/2132771434,v1:127.0.0.1:6837/2132771434] exists,up ddf62834-f7d1-48cf-a44f-d1907112ade4 2026-02-18T00:43:40.261 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:43:40.261 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:43:40.261 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:43:40.261 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-18T00:43:40.262 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 5 2026-02-18T00:43:40.262 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:43:40.262 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:43:40.262 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=5 2026-02-18T00:43:40.262 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:43:40.262 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/5 2026-02-18T00:43:40.262 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:43:40.262 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:43:40.262 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:43:40.262 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:43:40.262 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/5' 2026-02-18T00:43:40.262 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/5/journal' 2026-02-18T00:43:40.263 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:43:40.263 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:43:40.263 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2026-02-18T00:43:40.263 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:43:40.263 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:43:40.263 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:43:40.265 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:43:40.265 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:40.265 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:40.265 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:40.265 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:40.265 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:43:40.265 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:43:40.265 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:43:40.265 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2026-02-18T00:43:40.265 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2026-02-18T00:43:40.266 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:43:40.266 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:43:40.266 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:43:40.266 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:43:40.266 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:43:40.266 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:43:40.266 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/5 2026-02-18T00:43:40.266 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:43:40.267 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=9d261719-88cd-4f98-8190-8a0e3e36d705 2026-02-18T00:43:40.267 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd5 9d261719-88cd-4f98-8190-8a0e3e36d705' 2026-02-18T00:43:40.267 INFO:tasks.workunit.client.0.trial038.stdout:add osd5 9d261719-88cd-4f98-8190-8a0e3e36d705 2026-02-18T00:43:40.267 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:43:40.278 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQA8C5VpTYh3EBAArr3kk+bZpyk2tIG9LdLdiw== 2026-02-18T00:43:40.278 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQA8C5VpTYh3EBAArr3kk+bZpyk2tIG9LdLdiw=="}' 2026-02-18T00:43:40.278 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 9d261719-88cd-4f98-8190-8a0e3e36d705 -i td/mon-stretch-fail-recovery/5/new.json 2026-02-18T00:43:40.492 INFO:tasks.workunit.client.0.trial038.stdout:5 2026-02-18T00:43:40.502 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/5/new.json 2026-02-18T00:43:40.502 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 5 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/5 --osd-journal=td/mon-stretch-fail-recovery/5/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQA8C5VpTYh3EBAArr3kk+bZpyk2tIG9LdLdiw== --osd-uuid 9d261719-88cd-4f98-8190-8a0e3e36d705 2026-02-18T00:43:40.519 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:40.511+0000 7fc80889ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:40.520 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:40.515+0000 7fc80889ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:40.521 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:40.515+0000 7fc80889ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:40.522 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:40.515+0000 7fc80889ca40 -1 bdev(0x55d415b94800 td/mon-stretch-fail-recovery/5/block) open stat got: (1) Operation not permitted 2026-02-18T00:43:40.522 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:40.515+0000 7fc80889ca40 -1 bluestore(td/mon-stretch-fail-recovery/5) _read_fsid unparsable uuid 2026-02-18T00:43:41.197 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/5/keyring 2026-02-18T00:43:41.197 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:43:41.198 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd5 key to auth repository 2026-02-18T00:43:41.199 INFO:tasks.workunit.client.0.trial038.stdout:adding osd5 key to auth repository 2026-02-18T00:43:41.199 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/5/keyring auth add osd.5 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:43:41.487 INFO:tasks.workunit.client.0.trial038.stdout:start osd.5 2026-02-18T00:43:41.488 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.5 2026-02-18T00:43:41.488 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 5 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/5 --osd-journal=td/mon-stretch-fail-recovery/5/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:43:41.489 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:43:41.489 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:43:41.489 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:43:41.509 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:41.503+0000 7fc2e553ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:41.510 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:41.503+0000 7fc2e553ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:41.511 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:41.507+0000 7fc2e553ba40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:41.701 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 5 2026-02-18T00:43:41.701 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:43:41.701 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=5 2026-02-18T00:43:41.701 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:43:41.702 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:43:41.702 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:41.702 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:43:41.703 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:43:41.703 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:41.703 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-18T00:43:41.918 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:43:41.937 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:41.931+0000 7fc2e553ba40 -1 Falling back to public interface 2026-02-18T00:43:42.562 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:42.555+0000 7fc2e553ba40 -1 osd.5 0 log_to_monitors true 2026-02-18T00:43:42.920 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:43:42.920 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:42.920 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-18T00:43:42.921 INFO:tasks.workunit.client.0.trial038.stdout:1 2026-02-18T00:43:42.921 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:42.921 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-18T00:43:43.142 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:43:44.143 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:43:44.144 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:44.144 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-18T00:43:44.144 INFO:tasks.workunit.client.0.trial038.stdout:2 2026-02-18T00:43:44.145 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:44.145 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.5 up' 2026-02-18T00:43:44.357 INFO:tasks.workunit.client.0.trial038.stdout:osd.5 up in weight 1 up_from 25 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/1439693207,v1:127.0.0.1:6843/1439693207] [v2:127.0.0.1:6844/1439693207,v1:127.0.0.1:6845/1439693207] exists,up 9d261719-88cd-4f98-8190-8a0e3e36d705 2026-02-18T00:43:44.360 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:43:44.360 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:43:44.360 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:43:44.361 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-18T00:43:44.361 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 6 2026-02-18T00:43:44.361 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:43:44.361 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:43:44.361 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=6 2026-02-18T00:43:44.361 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:43:44.361 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/6 2026-02-18T00:43:44.361 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:43:44.361 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:43:44.361 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:43:44.361 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:43:44.362 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/6' 2026-02-18T00:43:44.362 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/6/journal' 2026-02-18T00:43:44.362 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:43:44.362 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:43:44.362 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2026-02-18T00:43:44.362 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:43:44.362 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:43:44.362 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:43:44.362 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:43:44.362 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:44.362 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:44.362 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:44.363 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:44.363 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:43:44.363 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:43:44.363 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:43:44.363 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2026-02-18T00:43:44.363 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2026-02-18T00:43:44.363 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:43:44.363 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:43:44.363 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:43:44.363 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:43:44.363 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:43:44.363 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:43:44.364 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/6 2026-02-18T00:43:44.364 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:43:44.364 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=228ea7f8-84eb-476b-a028-ef6520db7d78 2026-02-18T00:43:44.365 INFO:tasks.workunit.client.0.trial038.stdout:add osd6 228ea7f8-84eb-476b-a028-ef6520db7d78 2026-02-18T00:43:44.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd6 228ea7f8-84eb-476b-a028-ef6520db7d78' 2026-02-18T00:43:44.365 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:43:44.374 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBAC5VpptgtFhAAPNJd5FxTfaEbg5E7WzR3oA== 2026-02-18T00:43:44.374 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBAC5VpptgtFhAAPNJd5FxTfaEbg5E7WzR3oA=="}' 2026-02-18T00:43:44.374 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 228ea7f8-84eb-476b-a028-ef6520db7d78 -i td/mon-stretch-fail-recovery/6/new.json 2026-02-18T00:43:44.586 INFO:tasks.workunit.client.0.trial038.stdout:6 2026-02-18T00:43:44.597 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/6/new.json 2026-02-18T00:43:44.598 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 6 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/6 --osd-journal=td/mon-stretch-fail-recovery/6/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBAC5VpptgtFhAAPNJd5FxTfaEbg5E7WzR3oA== --osd-uuid 228ea7f8-84eb-476b-a028-ef6520db7d78 2026-02-18T00:43:44.612 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:44.607+0000 7f35f387aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:44.614 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:44.607+0000 7f35f387aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:44.615 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:44.611+0000 7f35f387aa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:44.616 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:44.611+0000 7f35f387aa40 -1 bdev(0x55a596d6c800 td/mon-stretch-fail-recovery/6/block) open stat got: (1) Operation not permitted 2026-02-18T00:43:44.616 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:44.611+0000 7f35f387aa40 -1 bluestore(td/mon-stretch-fail-recovery/6) _read_fsid unparsable uuid 2026-02-18T00:43:45.242 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/6/keyring 2026-02-18T00:43:45.243 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:43:45.244 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd6 key to auth repository 2026-02-18T00:43:45.244 INFO:tasks.workunit.client.0.trial038.stdout:adding osd6 key to auth repository 2026-02-18T00:43:45.244 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/6/keyring auth add osd.6 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:43:45.526 INFO:tasks.workunit.client.0.trial038.stdout:start osd.6 2026-02-18T00:43:45.527 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.6 2026-02-18T00:43:45.527 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 6 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/6 --osd-journal=td/mon-stretch-fail-recovery/6/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:43:45.527 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:43:45.527 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:43:45.527 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:43:45.548 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:45.543+0000 7efc8af1da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:45.550 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:45.543+0000 7efc8af1da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:45.551 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:45.547+0000 7efc8af1da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:45.849 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:45.843+0000 7efc8af1da40 -1 Falling back to public interface 2026-02-18T00:43:46.421 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:46.415+0000 7efc8af1da40 -1 osd.6 0 log_to_monitors true 2026-02-18T00:43:48.749 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 6 2026-02-18T00:43:48.749 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:43:48.749 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=6 2026-02-18T00:43:48.749 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:43:48.749 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:43:48.750 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:48.750 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:43:48.750 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:43:48.750 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:48.751 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.6 up' 2026-02-18T00:43:48.968 INFO:tasks.workunit.client.0.trial038.stdout:osd.6 up in weight 1 up_from 29 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/4242866314,v1:127.0.0.1:6851/4242866314] [v2:127.0.0.1:6852/4242866314,v1:127.0.0.1:6853/4242866314] exists,up 228ea7f8-84eb-476b-a028-ef6520db7d78 2026-02-18T00:43:48.972 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:43:48.972 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:43:48.972 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:43:48.972 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:57: TEST_stretched_cluster_failover_add_three_osds: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-18T00:43:48.973 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:59: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 7 2026-02-18T00:43:48.973 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:43:48.973 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:43:48.973 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=7 2026-02-18T00:43:48.973 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:43:48.973 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/7 2026-02-18T00:43:48.973 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:43:48.973 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:43:48.973 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:43:48.973 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:43:48.973 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/7' 2026-02-18T00:43:48.973 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/7/journal' 2026-02-18T00:43:48.974 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:43:48.974 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:43:48.974 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2026-02-18T00:43:48.974 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:43:48.974 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:43:48.974 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:43:48.974 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:43:48.974 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:43:48.974 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:43:48.974 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:48.974 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:43:48.974 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:43:48.974 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:43:48.975 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:43:48.975 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2026-02-18T00:43:48.975 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2026-02-18T00:43:48.975 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:43:48.975 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:43:48.975 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:43:48.975 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:43:48.975 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:43:48.975 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:43:48.975 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/7 2026-02-18T00:43:48.976 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:43:48.976 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=26014c07-695e-4fce-8d58-837a0bf024ae 2026-02-18T00:43:48.976 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd7 26014c07-695e-4fce-8d58-837a0bf024ae' 2026-02-18T00:43:48.977 INFO:tasks.workunit.client.0.trial038.stdout:add osd7 26014c07-695e-4fce-8d58-837a0bf024ae 2026-02-18T00:43:48.977 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:43:48.985 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQBEC5VpguylOhAADAKPXr7fLG5G4wUzeuHw/g== 2026-02-18T00:43:48.985 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQBEC5VpguylOhAADAKPXr7fLG5G4wUzeuHw/g=="}' 2026-02-18T00:43:48.986 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 26014c07-695e-4fce-8d58-837a0bf024ae -i td/mon-stretch-fail-recovery/7/new.json 2026-02-18T00:43:52.212 INFO:tasks.workunit.client.0.trial038.stdout:7 2026-02-18T00:43:52.222 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/7/new.json 2026-02-18T00:43:52.224 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 7 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/7 --osd-journal=td/mon-stretch-fail-recovery/7/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQBEC5VpguylOhAADAKPXr7fLG5G4wUzeuHw/g== --osd-uuid 26014c07-695e-4fce-8d58-837a0bf024ae 2026-02-18T00:43:52.245 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:52.239+0000 7fbbd7deea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:52.247 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:52.243+0000 7fbbd7deea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:52.248 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:52.243+0000 7fbbd7deea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:52.249 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:52.243+0000 7fbbd7deea40 -1 bdev(0x56011f13c800 td/mon-stretch-fail-recovery/7/block) open stat got: (1) Operation not permitted 2026-02-18T00:43:52.249 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:52.243+0000 7fbbd7deea40 -1 bluestore(td/mon-stretch-fail-recovery/7) _read_fsid unparsable uuid 2026-02-18T00:43:52.911 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/7/keyring 2026-02-18T00:43:52.911 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:43:52.912 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd7 key to auth repository 2026-02-18T00:43:52.912 INFO:tasks.workunit.client.0.trial038.stdout:adding osd7 key to auth repository 2026-02-18T00:43:52.912 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/7/keyring auth add osd.7 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:43:53.197 INFO:tasks.workunit.client.0.trial038.stdout:start osd.7 2026-02-18T00:43:53.198 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.7 2026-02-18T00:43:53.199 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 7 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/7 --osd-journal=td/mon-stretch-fail-recovery/7/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:43:53.199 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:43:53.199 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:43:53.199 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:43:53.213 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:53.207+0000 7fa84213ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:53.215 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:53.207+0000 7fa84213ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:53.216 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:53.211+0000 7fa84213ca40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:43:53.537 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:53.531+0000 7fa84213ca40 -1 Falling back to public interface 2026-02-18T00:43:54.173 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:43:54.167+0000 7fa84213ca40 -1 osd.7 0 log_to_monitors true 2026-02-18T00:43:56.415 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 7 2026-02-18T00:43:56.416 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:43:56.416 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=7 2026-02-18T00:43:56.416 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:43:56.416 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:43:56.416 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:43:56.416 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:43:56.416 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:43:56.416 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:43:56.416 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.7 up' 2026-02-18T00:43:56.622 INFO:tasks.workunit.client.0.trial038.stdout:osd.7 up in weight 1 up_from 33 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6858/1455971884,v1:127.0.0.1:6859/1455971884] [v2:127.0.0.1:6860/1455971884,v1:127.0.0.1:6861/1455971884] exists,up 26014c07-695e-4fce-8d58-837a0bf024ae 2026-02-18T00:43:56.623 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:43:56.623 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:43:56.623 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:43:56.623 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:62: TEST_stretched_cluster_failover_add_three_osds: for zone in iris pze 2026-02-18T00:43:56.623 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:64: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush add-bucket iris zone 2026-02-18T00:43:56.863 INFO:tasks.workunit.client.0.trial038.stderr:bucket 'iris' already exists 2026-02-18T00:43:56.872 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:65: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move iris root=default 2026-02-18T00:43:57.142 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id -5 name 'iris' to location {root=default} in crush map 2026-02-18T00:43:57.152 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:62: TEST_stretched_cluster_failover_add_three_osds: for zone in iris pze 2026-02-18T00:43:57.152 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:64: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush add-bucket pze zone 2026-02-18T00:43:57.420 INFO:tasks.workunit.client.0.trial038.stderr:bucket 'pze' already exists 2026-02-18T00:43:57.430 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:65: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move pze root=default 2026-02-18T00:43:57.729 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id -7 name 'pze' to location {root=default} in crush map 2026-02-18T00:43:57.740 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:69: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush add-bucket node-2 host 2026-02-18T00:43:58.008 INFO:tasks.workunit.client.0.trial038.stderr:bucket 'node-2' already exists 2026-02-18T00:43:58.018 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:70: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush add-bucket node-3 host 2026-02-18T00:43:58.294 INFO:tasks.workunit.client.0.trial038.stderr:bucket 'node-3' already exists 2026-02-18T00:43:58.304 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:71: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush add-bucket node-4 host 2026-02-18T00:43:58.570 INFO:tasks.workunit.client.0.trial038.stderr:bucket 'node-4' already exists 2026-02-18T00:43:58.582 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:72: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush add-bucket node-5 host 2026-02-18T00:43:58.839 INFO:tasks.workunit.client.0.trial038.stderr:bucket 'node-5' already exists 2026-02-18T00:43:58.849 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:74: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move node-2 zone=iris 2026-02-18T00:43:59.119 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id -9 name 'node-2' to location {zone=iris} in crush map 2026-02-18T00:43:59.129 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:75: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move node-3 zone=iris 2026-02-18T00:43:59.402 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id -10 name 'node-3' to location {zone=iris} in crush map 2026-02-18T00:43:59.413 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:76: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move node-4 zone=pze 2026-02-18T00:43:59.709 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id -11 name 'node-4' to location {zone=pze} in crush map 2026-02-18T00:43:59.719 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:77: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move node-5 zone=pze 2026-02-18T00:43:59.974 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id -12 name 'node-5' to location {zone=pze} in crush map 2026-02-18T00:43:59.984 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:79: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.0 host=node-2 2026-02-18T00:44:00.250 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id 0 name 'osd.0' to location {host=node-2} in crush map 2026-02-18T00:44:00.260 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:80: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.1 host=node-2 2026-02-18T00:44:00.527 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id 1 name 'osd.1' to location {host=node-2} in crush map 2026-02-18T00:44:00.537 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:81: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.2 host=node-3 2026-02-18T00:44:00.812 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id 2 name 'osd.2' to location {host=node-3} in crush map 2026-02-18T00:44:00.822 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:82: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.3 host=node-3 2026-02-18T00:44:01.090 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id 3 name 'osd.3' to location {host=node-3} in crush map 2026-02-18T00:44:01.100 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:83: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.4 host=node-4 2026-02-18T00:44:01.370 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id 4 name 'osd.4' to location {host=node-4} in crush map 2026-02-18T00:44:01.380 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:84: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.5 host=node-4 2026-02-18T00:44:01.678 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id 5 name 'osd.5' to location {host=node-4} in crush map 2026-02-18T00:44:01.688 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:85: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.6 host=node-5 2026-02-18T00:44:01.963 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id 6 name 'osd.6' to location {host=node-5} in crush map 2026-02-18T00:44:01.974 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:86: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush move osd.7 host=node-5 2026-02-18T00:44:02.239 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id 7 name 'osd.7' to location {host=node-5} in crush map 2026-02-18T00:44:02.249 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:88: TEST_stretched_cluster_failover_add_three_osds: ceph mon set_location a zone=iris host=node-2 2026-02-18T00:44:07.562 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:89: TEST_stretched_cluster_failover_add_three_osds: ceph mon set_location b zone=iris host=node-3 2026-02-18T00:44:07.843 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:90: TEST_stretched_cluster_failover_add_three_osds: ceph mon set_location c zone=pze host=node-4 2026-02-18T00:44:14.143 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:91: TEST_stretched_cluster_failover_add_three_osds: ceph mon set_location d zone=pze host=node-5 2026-02-18T00:44:25.425 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:93: TEST_stretched_cluster_failover_add_three_osds: hostname -s 2026-02-18T00:44:25.426 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:93: TEST_stretched_cluster_failover_add_three_osds: hostname=trial038 2026-02-18T00:44:25.426 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:94: TEST_stretched_cluster_failover_add_three_osds: ceph osd crush remove trial038 2026-02-18T00:44:25.672 INFO:tasks.workunit.client.0.trial038.stderr:device 'trial038' does not appear in the crush map 2026-02-18T00:44:25.683 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:95: TEST_stretched_cluster_failover_add_three_osds: ceph osd getcrushmap 2026-02-18T00:44:25.892 INFO:tasks.workunit.client.0.trial038.stderr:38 2026-02-18T00:44:25.903 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:96: TEST_stretched_cluster_failover_add_three_osds: crushtool --decompile crushmap 2026-02-18T00:44:25.915 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:97: TEST_stretched_cluster_failover_add_three_osds: sed 's/^# end crush map$//' crushmap.txt 2026-02-18T00:44:25.916 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:98: TEST_stretched_cluster_failover_add_three_osds: cat 2026-02-18T00:44:25.917 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:115: TEST_stretched_cluster_failover_add_three_osds: crushtool --compile crushmap_modified.txt -o crushmap.bin 2026-02-18T00:44:25.926 INFO:tasks.workunit.client.0.trial038.stderr:WARNING: min_size is no longer supported, ignoring 2026-02-18T00:44:25.927 INFO:tasks.workunit.client.0.trial038.stderr:WARNING: max_size is no longer supported, ignoring 2026-02-18T00:44:25.927 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:116: TEST_stretched_cluster_failover_add_three_osds: ceph osd setcrushmap -i crushmap.bin 2026-02-18T00:44:26.205 INFO:tasks.workunit.client.0.trial038.stderr:40 2026-02-18T00:44:26.216 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:117: TEST_stretched_cluster_failover_add_three_osds: local stretched_poolname=stretched_rbdpool 2026-02-18T00:44:26.216 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:118: TEST_stretched_cluster_failover_add_three_osds: ceph osd pool create stretched_rbdpool 32 32 stretch_rule 2026-02-18T00:44:26.465 INFO:tasks.workunit.client.0.trial038.stderr:pool 'stretched_rbdpool' already exists 2026-02-18T00:44:26.476 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:119: TEST_stretched_cluster_failover_add_three_osds: ceph osd pool set stretched_rbdpool size 4 2026-02-18T00:44:26.843 INFO:tasks.workunit.client.0.trial038.stderr:set pool 1 size to 4 2026-02-18T00:44:26.855 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:121: TEST_stretched_cluster_failover_add_three_osds: sleep 3 2026-02-18T00:44:29.856 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:123: TEST_stretched_cluster_failover_add_three_osds: ceph mon set_location e zone=arbiter host=node-1 2026-02-18T00:44:30.154 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:124: TEST_stretched_cluster_failover_add_three_osds: ceph mon enable_stretch_mode e stretch_rule zone 2026-02-18T00:44:41.464 INFO:tasks.workunit.client.0.trial038.stderr:Second attempt of previously successful command failed with EINVAL: stretch mode is already engaged 2026-02-18T00:44:41.464 INFO:tasks.workunit.client.0.trial038.stderr:stretch mode is already engaged 2026-02-18T00:44:41.477 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:126: TEST_stretched_cluster_failover_add_three_osds: kill_daemons td/mon-stretch-fail-recovery KILL mon.c 2026-02-18T00:44:41.478 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:44:41.478 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:44:41.478 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:44:41.478 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:44:41.478 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:44:41.586 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:44:41.586 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:127: TEST_stretched_cluster_failover_add_three_osds: kill_daemons td/mon-stretch-fail-recovery KILL mon.d 2026-02-18T00:44:41.587 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:44:41.587 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:44:41.587 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:44:41.587 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:44:41.588 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:44:41.694 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:44:41.694 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:129: TEST_stretched_cluster_failover_add_three_osds: kill_daemons td/mon-stretch-fail-recovery KILL osd.4 2026-02-18T00:44:41.694 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:44:41.694 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:44:41.695 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:44:41.695 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:44:41.695 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:44:41.801 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:44:41.801 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:130: TEST_stretched_cluster_failover_add_three_osds: kill_daemons td/mon-stretch-fail-recovery KILL osd.5 2026-02-18T00:44:41.801 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:44:41.801 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:44:41.802 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:44:41.803 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:44:41.803 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:44:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:44:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:131: TEST_stretched_cluster_failover_add_three_osds: kill_daemons td/mon-stretch-fail-recovery KILL osd.6 2026-02-18T00:44:41.909 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:44:41.909 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:44:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:44:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:44:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:44:42.017 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:44:42.018 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:132: TEST_stretched_cluster_failover_add_three_osds: kill_daemons td/mon-stretch-fail-recovery KILL osd.7 2026-02-18T00:44:42.018 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:44:42.018 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:44:42.018 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:44:42.018 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:44:42.018 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:44:42.124 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:44:42.125 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:134: TEST_stretched_cluster_failover_add_three_osds: ceph -s 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: cluster: 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: id: e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: stretch_mode: ENABLED 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: health: HEALTH_OK 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: services: 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: mon: 5 daemons, quorum a,b,c,d,e (age 0.958135s) [leader: a] 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: mgr: x(active, since 87s), standbys: y, z 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: osd: 8 osds: 8 up (since 47s), 8 in (since 50s) 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: data: 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: pools: 1 pools, 32 pgs 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: objects: 0 objects, 0 B 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: usage: 215 MiB used, 800 GiB / 800 GiB avail 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: pgs: 34.375% pgs unknown 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: 21 active+clean 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: 11 unknown 2026-02-18T00:44:42.395 INFO:tasks.workunit.client.0.trial038.stdout: 2026-02-18T00:44:42.404 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:136: TEST_stretched_cluster_failover_add_three_osds: sleep 3 2026-02-18T00:44:45.407 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:138: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 8 2026-02-18T00:44:45.408 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:44:45.408 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:44:45.408 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=8 2026-02-18T00:44:45.408 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:44:45.408 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/8 2026-02-18T00:44:45.408 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:44:45.408 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:44:45.408 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:44:45.409 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:44:45.409 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/8' 2026-02-18T00:44:45.409 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/8/journal' 2026-02-18T00:44:45.409 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:44:45.409 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:44:45.409 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2026-02-18T00:44:45.409 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:44:45.409 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:44:45.409 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:44:45.409 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:44:45.409 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:44:45.410 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:44:45.410 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:44:45.411 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:44:45.411 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:44:45.411 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:44:45.411 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:44:45.411 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2026-02-18T00:44:45.411 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2026-02-18T00:44:45.412 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:44:45.412 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:44:45.412 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:44:45.412 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:44:45.412 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:44:45.412 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:44:45.412 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/8 2026-02-18T00:44:45.412 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:44:45.413 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=1b58dbce-4492-46b3-9d58-d2d00d204f06 2026-02-18T00:44:45.413 INFO:tasks.workunit.client.0.trial038.stdout:add osd8 1b58dbce-4492-46b3-9d58-d2d00d204f06 2026-02-18T00:44:45.413 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd8 1b58dbce-4492-46b3-9d58-d2d00d204f06' 2026-02-18T00:44:45.414 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:44:45.425 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQB9C5VpG0k8GRAA7LU0aHr+C0QG6j8j2jmPcQ== 2026-02-18T00:44:45.425 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQB9C5VpG0k8GRAA7LU0aHr+C0QG6j8j2jmPcQ=="}' 2026-02-18T00:44:45.425 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 1b58dbce-4492-46b3-9d58-d2d00d204f06 -i td/mon-stretch-fail-recovery/8/new.json 2026-02-18T00:45:01.623 INFO:tasks.workunit.client.0.trial038.stdout:8 2026-02-18T00:45:01.634 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/8/new.json 2026-02-18T00:45:01.636 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 8 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/8 --osd-journal=td/mon-stretch-fail-recovery/8/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQB9C5VpG0k8GRAA7LU0aHr+C0QG6j8j2jmPcQ== --osd-uuid 1b58dbce-4492-46b3-9d58-d2d00d204f06 2026-02-18T00:45:01.659 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:01.653+0000 7f447f127a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:10.663 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:10.661+0000 7f447f127a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:10.665 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:10.661+0000 7f447f127a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:10.666 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:10.661+0000 7f447f127a40 -1 bdev(0x55895b3a2c00 td/mon-stretch-fail-recovery/8/block) open stat got: (1) Operation not permitted 2026-02-18T00:45:10.666 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:10.661+0000 7f447f127a40 -1 bluestore(td/mon-stretch-fail-recovery/8) _read_fsid unparsable uuid 2026-02-18T00:45:11.319 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/8/keyring 2026-02-18T00:45:11.319 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:45:11.320 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd8 key to auth repository 2026-02-18T00:45:11.320 INFO:tasks.workunit.client.0.trial038.stdout:adding osd8 key to auth repository 2026-02-18T00:45:11.320 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/8/keyring auth add osd.8 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:45:11.611 INFO:tasks.workunit.client.0.trial038.stdout:start osd.8 2026-02-18T00:45:11.612 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.8 2026-02-18T00:45:11.612 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 8 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/8 --osd-journal=td/mon-stretch-fail-recovery/8/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:45:11.613 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:45:11.613 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:45:11.613 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:45:11.633 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:11.629+0000 7f94cb2c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:11.635 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:11.633+0000 7f94cb2c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:11.636 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:11.633+0000 7f94cb2c0a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:11.992 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:11.989+0000 7f94cb2c0a40 -1 Falling back to public interface 2026-02-18T00:45:12.655 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:12.649+0000 7f94cb2c0a40 -1 osd.8 0 log_to_monitors true 2026-02-18T00:45:14.826 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 8 2026-02-18T00:45:14.826 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:45:14.826 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=8 2026-02-18T00:45:14.826 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:45:14.826 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:45:14.826 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:45:14.827 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:45:14.827 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:45:14.828 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:45:14.828 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.8 up' 2026-02-18T00:45:15.036 INFO:tasks.workunit.client.0.trial038.stdout:osd.8 up in weight 1 up_from 70 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6834/3673783829,v1:127.0.0.1:6835/3673783829] [v2:127.0.0.1:6836/3673783829,v1:127.0.0.1:6837/3673783829] exists,up 1b58dbce-4492-46b3-9d58-d2d00d204f06 2026-02-18T00:45:15.039 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:45:15.039 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:45:15.039 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:45:15.039 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:139: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 9 2026-02-18T00:45:15.039 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:45:15.039 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:45:15.040 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=9 2026-02-18T00:45:15.040 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:45:15.040 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/9 2026-02-18T00:45:15.040 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:45:15.040 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:45:15.040 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:45:15.040 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:45:15.040 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/9' 2026-02-18T00:45:15.040 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/9/journal' 2026-02-18T00:45:15.040 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:45:15.040 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:45:15.040 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2026-02-18T00:45:15.041 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:45:15.041 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:45:15.041 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:45:15.041 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:45:15.041 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:15.041 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:45:15.041 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:45:15.041 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:45:15.041 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:45:15.041 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:45:15.041 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:45:15.041 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2026-02-18T00:45:15.042 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2026-02-18T00:45:15.042 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:45:15.042 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:45:15.042 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:45:15.042 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:45:15.042 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:45:15.042 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:45:15.042 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/9 2026-02-18T00:45:15.043 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:45:15.043 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=307d0f5e-459d-47f6-bc35-cf7970371336 2026-02-18T00:45:15.043 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd9 307d0f5e-459d-47f6-bc35-cf7970371336' 2026-02-18T00:45:15.043 INFO:tasks.workunit.client.0.trial038.stdout:add osd9 307d0f5e-459d-47f6-bc35-cf7970371336 2026-02-18T00:45:15.044 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:45:15.052 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCbC5VpV10MAxAACNOAe9OdJyKGUHdNwXxMkw== 2026-02-18T00:45:15.052 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCbC5VpV10MAxAACNOAe9OdJyKGUHdNwXxMkw=="}' 2026-02-18T00:45:15.052 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 307d0f5e-459d-47f6-bc35-cf7970371336 -i td/mon-stretch-fail-recovery/9/new.json 2026-02-18T00:45:15.271 INFO:tasks.workunit.client.0.trial038.stdout:9 2026-02-18T00:45:15.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/9/new.json 2026-02-18T00:45:15.282 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 9 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/9 --osd-journal=td/mon-stretch-fail-recovery/9/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCbC5VpV10MAxAACNOAe9OdJyKGUHdNwXxMkw== --osd-uuid 307d0f5e-459d-47f6-bc35-cf7970371336 2026-02-18T00:45:15.298 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:15.292+0000 7f3e8b71ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:15.300 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:15.296+0000 7f3e8b71ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:15.301 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:15.296+0000 7f3e8b71ea40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:15.301 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:15.296+0000 7f3e8b71ea40 -1 bdev(0x557997452800 td/mon-stretch-fail-recovery/9/block) open stat got: (1) Operation not permitted 2026-02-18T00:45:15.301 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:15.296+0000 7f3e8b71ea40 -1 bluestore(td/mon-stretch-fail-recovery/9) _read_fsid unparsable uuid 2026-02-18T00:45:15.986 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/9/keyring 2026-02-18T00:45:15.986 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:45:15.987 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd9 key to auth repository 2026-02-18T00:45:15.987 INFO:tasks.workunit.client.0.trial038.stdout:adding osd9 key to auth repository 2026-02-18T00:45:15.987 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/9/keyring auth add osd.9 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:45:16.277 INFO:tasks.workunit.client.0.trial038.stdout:start osd.9 2026-02-18T00:45:16.278 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.9 2026-02-18T00:45:16.278 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 9 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/9 --osd-journal=td/mon-stretch-fail-recovery/9/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:45:16.278 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:45:16.278 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:45:16.278 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:45:16.292 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:16.288+0000 7f3db8a02a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:16.294 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:16.288+0000 7f3db8a02a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:16.295 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:16.292+0000 7f3db8a02a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:16.608 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:16.604+0000 7f3db8a02a40 -1 Falling back to public interface 2026-02-18T00:45:17.205 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:17.200+0000 7f3db8a02a40 -1 osd.9 0 log_to_monitors true 2026-02-18T00:45:19.497 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 9 2026-02-18T00:45:19.498 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:45:19.498 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=9 2026-02-18T00:45:19.498 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:45:19.498 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:45:19.498 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:45:19.499 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:45:19.499 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:45:19.499 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:45:19.499 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-18T00:45:19.713 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:45:20.715 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:45:20.715 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:45:20.716 INFO:tasks.workunit.client.0.trial038.stdout:1 2026-02-18T00:45:20.716 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-18T00:45:20.716 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:45:20.717 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.9 up' 2026-02-18T00:45:20.921 INFO:tasks.workunit.client.0.trial038.stdout:osd.9 up in weight 1 up_from 74 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6842/498055880,v1:127.0.0.1:6843/498055880] [v2:127.0.0.1:6844/498055880,v1:127.0.0.1:6845/498055880] exists,up 307d0f5e-459d-47f6-bc35-cf7970371336 2026-02-18T00:45:20.925 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:45:20.925 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:45:20.925 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:45:20.925 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:140: TEST_stretched_cluster_failover_add_three_osds: run_osd td/mon-stretch-fail-recovery 10 2026-02-18T00:45:20.925 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:45:20.926 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:45:20.926 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=10 2026-02-18T00:45:20.926 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:45:20.926 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretch-fail-recovery/10 2026-02-18T00:45:20.926 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:45:20.926 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:45:20.926 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:45:20.926 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:45:20.926 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretch-fail-recovery/10' 2026-02-18T00:45:20.926 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretch-fail-recovery/10/journal' 2026-02-18T00:45:20.926 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:45:20.927 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:45:20.927 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretch-fail-recovery' 2026-02-18T00:45:20.927 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:45:20.927 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:45:20.927 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:45:20.927 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:45:20.927 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:20.927 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:45:20.927 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:45:20.927 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' 2026-02-18T00:45:20.928 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:45:20.928 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:45:20.928 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:45:20.928 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretch-fail-recovery/$name.log' 2026-02-18T00:45:20.928 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretch-fail-recovery/$name.pid' 2026-02-18T00:45:20.928 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:45:20.928 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:45:20.928 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:45:20.928 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:45:20.928 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:45:20.928 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:45:20.929 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretch-fail-recovery/10 2026-02-18T00:45:20.929 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:45:20.929 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=f1154d3a-4520-4abd-af33-ec548adda023 2026-02-18T00:45:20.930 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd10 f1154d3a-4520-4abd-af33-ec548adda023' 2026-02-18T00:45:20.930 INFO:tasks.workunit.client.0.trial038.stdout:add osd10 f1154d3a-4520-4abd-af33-ec548adda023 2026-02-18T00:45:20.930 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:45:20.937 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQCgC5VpWBzRNxAAC4hvHYTl9WvH1+IzVySPxg== 2026-02-18T00:45:20.938 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQCgC5VpWBzRNxAAC4hvHYTl9WvH1+IzVySPxg=="}' 2026-02-18T00:45:20.938 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new f1154d3a-4520-4abd-af33-ec548adda023 -i td/mon-stretch-fail-recovery/10/new.json 2026-02-18T00:45:21.164 INFO:tasks.workunit.client.0.trial038.stdout:10 2026-02-18T00:45:21.175 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretch-fail-recovery/10/new.json 2026-02-18T00:45:21.176 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 10 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/10 --osd-journal=td/mon-stretch-fail-recovery/10/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQCgC5VpWBzRNxAAC4hvHYTl9WvH1+IzVySPxg== --osd-uuid f1154d3a-4520-4abd-af33-ec548adda023 2026-02-18T00:45:21.200 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:21.196+0000 7feac17a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:21.201 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:21.196+0000 7feac17a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:21.203 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:21.196+0000 7feac17a7a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:21.203 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:21.200+0000 7feac17a7a40 -1 bdev(0x5633bb81c800 td/mon-stretch-fail-recovery/10/block) open stat got: (1) Operation not permitted 2026-02-18T00:45:21.203 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:21.200+0000 7feac17a7a40 -1 bluestore(td/mon-stretch-fail-recovery/10) _read_fsid unparsable uuid 2026-02-18T00:45:21.859 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretch-fail-recovery/10/keyring 2026-02-18T00:45:21.859 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:45:21.860 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd10 key to auth repository 2026-02-18T00:45:21.860 INFO:tasks.workunit.client.0.trial038.stdout:adding osd10 key to auth repository 2026-02-18T00:45:21.861 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretch-fail-recovery/10/keyring auth add osd.10 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:45:25.145 INFO:tasks.workunit.client.0.trial038.stdout:start osd.10 2026-02-18T00:45:25.146 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.10 2026-02-18T00:45:25.147 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 10 --fsid=e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretch-fail-recovery/10 --osd-journal=td/mon-stretch-fail-recovery/10/journal --chdir= --run-dir=td/mon-stretch-fail-recovery '--admin-socket=/tmp/ceph-asok.7520/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretch-fail-recovery/$name.log' '--pid-file=td/mon-stretch-fail-recovery/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:45:25.147 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:45:25.147 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:45:25.147 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:45:25.169 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:25.164+0000 7f303f45fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:25.364 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 10 2026-02-18T00:45:25.364 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:45:25.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=10 2026-02-18T00:45:25.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:45:25.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:45:25.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:45:25.365 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:45:25.365 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:45:25.366 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:45:25.366 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2026-02-18T00:45:28.173 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:28.168+0000 7f303f45fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:28.174 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:28.168+0000 7f303f45fa40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:45:28.529 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:28.524+0000 7f303f45fa40 -1 Falling back to public interface 2026-02-18T00:45:28.584 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:45:29.273 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:45:29.268+0000 7f303f45fa40 -1 osd.10 0 log_to_monitors true 2026-02-18T00:45:29.585 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:45:29.585 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:45:29.586 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-18T00:45:29.586 INFO:tasks.workunit.client.0.trial038.stdout:1 2026-02-18T00:45:29.586 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:45:29.587 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2026-02-18T00:45:29.800 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:45:30.801 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:45:30.802 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:45:30.802 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-18T00:45:30.802 INFO:tasks.workunit.client.0.trial038.stdout:2 2026-02-18T00:45:30.803 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:45:30.803 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2026-02-18T00:45:31.007 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:45:32.008 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:45:32.009 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:45:32.009 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 3 2026-02-18T00:45:32.009 INFO:tasks.workunit.client.0.trial038.stdout:3 2026-02-18T00:45:32.009 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:45:32.010 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.10 up' 2026-02-18T00:45:35.222 INFO:tasks.workunit.client.0.trial038.stdout:osd.10 up in weight 1 up_from 83 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6850/576725485,v1:127.0.0.1:6851/576725485] [v2:127.0.0.1:6852/576725485,v1:127.0.0.1:6853/576725485] exists,up f1154d3a-4520-4abd-af33-ec548adda023 2026-02-18T00:45:35.223 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:45:35.223 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:45:35.223 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:45:35.223 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:142: TEST_stretched_cluster_failover_add_three_osds: ceph -s 2026-02-18T00:45:38.493 INFO:tasks.workunit.client.0.trial038.stdout: cluster: 2026-02-18T00:45:38.494 INFO:tasks.workunit.client.0.trial038.stdout: id: e84dd4a4-dfcb-4bc3-9202-eab1d8d58faf 2026-02-18T00:45:38.494 INFO:tasks.workunit.client.0.trial038.stdout: stretch_mode: ENABLED 2026-02-18T00:45:38.494 INFO:tasks.workunit.client.0.trial038.stdout: health: HEALTH_WARN 2026-02-18T00:45:38.494 INFO:tasks.workunit.client.0.trial038.stdout: We are missing stretch mode buckets, only requiring 1 of 2 buckets to peer 2026-02-18T00:45:38.494 INFO:tasks.workunit.client.0.trial038.stdout: 2/5 mons down, quorum a,b,e 2026-02-18T00:45:38.494 INFO:tasks.workunit.client.0.trial038.stdout: 4 osds down 2026-02-18T00:45:38.494 INFO:tasks.workunit.client.0.trial038.stdout: 2 hosts (4 osds) down 2026-02-18T00:45:38.494 INFO:tasks.workunit.client.0.trial038.stdout: 1 zone (4 osds) down 2026-02-18T00:45:38.494 INFO:tasks.workunit.client.0.trial038.stdout: 2026-02-18T00:45:38.495 INFO:tasks.workunit.client.0.trial038.stdout: services: 2026-02-18T00:45:38.495 INFO:tasks.workunit.client.0.trial038.stdout: mon: 5 daemons, quorum a,b,e (age 31s) [leader: a], out of quorum: c, d 2026-02-18T00:45:38.495 INFO:tasks.workunit.client.0.trial038.stdout: mgr: x(active, since 2m), standbys: y, z 2026-02-18T00:45:38.495 INFO:tasks.workunit.client.0.trial038.stdout: osd: 11 osds: 7 up (since 5s), 11 in (since 17s) 2026-02-18T00:45:38.495 INFO:tasks.workunit.client.0.trial038.stdout: 2026-02-18T00:45:38.495 INFO:tasks.workunit.client.0.trial038.stdout: data: 2026-02-18T00:45:38.495 INFO:tasks.workunit.client.0.trial038.stdout: pools: 1 pools, 32 pgs 2026-02-18T00:45:38.495 INFO:tasks.workunit.client.0.trial038.stdout: objects: 0 objects, 0 B 2026-02-18T00:45:38.495 INFO:tasks.workunit.client.0.trial038.stdout: usage: 298 MiB used, 1.1 TiB / 1.1 TiB avail 2026-02-18T00:45:38.495 INFO:tasks.workunit.client.0.trial038.stdout: pgs: 32 active+undersized 2026-02-18T00:45:38.495 INFO:tasks.workunit.client.0.trial038.stdout: 2026-02-18T00:45:38.503 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:144: TEST_stretched_cluster_failover_add_three_osds: sleep 3 2026-02-18T00:45:41.506 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:146: TEST_stretched_cluster_failover_add_three_osds: teardown td/mon-stretch-fail-recovery 2026-02-18T00:45:41.506 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:45:41.506 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-18T00:45:41.506 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretch-fail-recovery KILL 2026-02-18T00:45:41.506 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:45:41.506 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:45:41.507 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:45:41.507 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:45:41.507 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:45:41.821 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:45:41.822 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-18T00:45:41.823 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-18T00:45:41.823 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-18T00:45:41.825 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-18T00:45:41.825 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-18T00:45:41.825 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-18T00:45:41.826 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:45:41.826 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-18T00:45:41.827 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-18T00:45:41.827 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:45:41.828 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-18T00:45:41.829 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-18T00:45:41.829 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretch-fail-recovery 2026-02-18T00:45:41.894 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-18T00:45:41.894 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:41.894 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:45:41.894 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.7520 2026-02-18T00:45:41.896 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-18T00:45:41.896 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-18T00:45:41.896 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-fail-recovery.sh:23: run: teardown td/mon-stretch-fail-recovery 2026-02-18T00:45:41.896 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:45:41.897 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-18T00:45:41.897 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretch-fail-recovery KILL 2026-02-18T00:45:41.897 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:45:41.897 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:45:41.897 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:45:41.897 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:45:41.897 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:45:41.898 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:45:41.899 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-18T00:45:41.899 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-18T00:45:41.900 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-18T00:45:41.901 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-18T00:45:41.901 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-18T00:45:41.902 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-18T00:45:41.902 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:45:41.902 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-18T00:45:41.903 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-18T00:45:41.903 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:45:41.904 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-18T00:45:41.905 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-18T00:45:41.905 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretch-fail-recovery 2026-02-18T00:45:41.906 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-18T00:45:41.906 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:41.907 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:45:41.907 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.7520 2026-02-18T00:45:41.908 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-18T00:45:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-18T00:45:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-18T00:45:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/mon-stretch-fail-recovery 0 2026-02-18T00:45:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretch-fail-recovery 2026-02-18T00:45:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-18T00:45:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretch-fail-recovery KILL 2026-02-18T00:45:41.909 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:45:41.909 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:45:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:45:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:45:41.909 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:45:41.911 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:45:41.911 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-18T00:45:41.912 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-18T00:45:41.912 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-18T00:45:41.913 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-18T00:45:41.914 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-18T00:45:41.914 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-18T00:45:41.915 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:45:41.915 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-18T00:45:41.916 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-18T00:45:41.916 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:45:41.916 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-18T00:45:41.917 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-18T00:45:41.917 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretch-fail-recovery 2026-02-18T00:45:41.919 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-18T00:45:41.919 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:41.919 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.7520 2026-02-18T00:45:41.919 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.7520 2026-02-18T00:45:41.921 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-18T00:45:41.922 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-18T00:45:41.922 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-18T00:45:41.922 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-18T00:45:41.923 DEBUG:teuthology.orchestra.run.trial038:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-18T00:45:41.972 INFO:tasks.workunit:Running workunit mon-stretch/mon-stretch-uneven-crush-weights.sh... 2026-02-18T00:45:41.972 DEBUG:teuthology.orchestra.run.trial038:workunit test mon-stretch/mon-stretch-uneven-crush-weights.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=adc1923395f2bcd6ae89b5d331d278364d1848ed TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh 2026-02-18T00:45:42.029 INFO:tasks.workunit.client.0.trial038.stderr:stty: 'standard input': Inappropriate ioctl for device 2026-02-18T00:45:42.032 INFO:tasks.workunit.client.0.trial038.stderr:+ PS4='${BASH_SOURCE[0]}:$LINENO: ${FUNCNAME[0]}: ' 2026-02-18T00:45:42.032 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: export PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-18T00:45:42.032 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2394: main: PATH=.:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/sbin 2026-02-18T00:45:42.033 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: export PYTHONWARNINGS=ignore 2026-02-18T00:45:42.033 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2395: main: PYTHONWARNINGS=ignore 2026-02-18T00:45:42.033 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: export CEPH_CONF=/dev/null 2026-02-18T00:45:42.033 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2396: main: CEPH_CONF=/dev/null 2026-02-18T00:45:42.033 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2397: main: unset CEPH_ARGS 2026-02-18T00:45:42.033 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2399: main: local code 2026-02-18T00:45:42.033 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2400: main: run td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.033 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:5: run: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.033 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:6: run: shift 2026-02-18T00:45:42.033 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:8: run: export CEPH_MON_A=127.0.0.1:7139 2026-02-18T00:45:42.033 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:8: run: CEPH_MON_A=127.0.0.1:7139 2026-02-18T00:45:42.034 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:9: run: export CEPH_MON_B=127.0.0.1:7141 2026-02-18T00:45:42.034 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:9: run: CEPH_MON_B=127.0.0.1:7141 2026-02-18T00:45:42.034 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:10: run: export CEPH_MON_C=127.0.0.1:7142 2026-02-18T00:45:42.034 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:10: run: CEPH_MON_C=127.0.0.1:7142 2026-02-18T00:45:42.034 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:11: run: export CEPH_MON_D=127.0.0.1:7143 2026-02-18T00:45:42.034 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:11: run: CEPH_MON_D=127.0.0.1:7143 2026-02-18T00:45:42.034 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:12: run: export CEPH_MON_E=127.0.0.1:7144 2026-02-18T00:45:42.034 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:12: run: CEPH_MON_E=127.0.0.1:7144 2026-02-18T00:45:42.034 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:13: run: export CEPH_ARGS 2026-02-18T00:45:42.034 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:14: run: uuidgen 2026-02-18T00:45:42.036 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:14: run: CEPH_ARGS+='--fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none ' 2026-02-18T00:45:42.037 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:16: run: export 'BASE_CEPH_ARGS=--fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none ' 2026-02-18T00:45:42.037 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:16: run: BASE_CEPH_ARGS='--fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none ' 2026-02-18T00:45:42.037 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:17: run: CEPH_ARGS+=--mon-host=127.0.0.1:7139 2026-02-18T00:45:42.037 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:19: run: set 2026-02-18T00:45:42.037 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:19: run: sed -n -e 's/^\(TEST_[0-9a-z_]*\) .*/\1/p' 2026-02-18T00:45:42.037 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:19: run: local funcs=TEST_stretched_cluster_uneven_weight 2026-02-18T00:45:42.037 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:20: run: for func in $funcs 2026-02-18T00:45:42.037 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:21: run: setup td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.037 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.037 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.038 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.038 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-18T00:45:42.038 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretched-cluster-uneven-weight KILL 2026-02-18T00:45:42.038 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:45:42.038 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:45:42.038 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:45:42.038 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:45:42.038 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:45:42.039 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:45:42.039 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-18T00:45:42.039 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-18T00:45:42.040 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-18T00:45:42.041 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-18T00:45:42.041 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-18T00:45:42.041 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-18T00:45:42.043 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:45:42.043 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-18T00:45:42.043 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-18T00:45:42.043 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:45:42.044 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-18T00:45:42.045 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-18T00:45:42.045 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.047 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-18T00:45:42.047 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:42.047 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:42.047 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.29825 2026-02-18T00:45:42.048 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-18T00:45:42.048 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-18T00:45:42.048 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.049 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-18T00:45:42.050 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:42.050 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:42.050 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.29825 2026-02-18T00:45:42.051 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-18T00:45:42.053 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 1024 -le 1024 ']' 2026-02-18T00:45:42.053 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:127: setup: ulimit -n 4096 2026-02-18T00:45:42.053 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-18T00:45:42.054 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mon-stretched-cluster-uneven-weight 1' TERM HUP INT 2026-02-18T00:45:42.054 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:22: run: TEST_stretched_cluster_uneven_weight td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.054 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:27: TEST_stretched_cluster_uneven_weight: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.054 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:28: TEST_stretched_cluster_uneven_weight: local OSDS=4 2026-02-18T00:45:42.054 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:29: TEST_stretched_cluster_uneven_weight: local weight=0.09000 2026-02-18T00:45:42.054 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:30: TEST_stretched_cluster_uneven_weight: setup td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.054 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:122: setup: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.054 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:123: setup: teardown td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.054 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.054 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-18T00:45:42.054 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretched-cluster-uneven-weight KILL 2026-02-18T00:45:42.055 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:45:42.055 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:45:42.055 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:45:42.055 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:45:42.055 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:45:42.055 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:45:42.056 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-18T00:45:42.056 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-18T00:45:42.057 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-18T00:45:42.057 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-18T00:45:42.058 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-18T00:45:42.058 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-18T00:45:42.059 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:45:42.059 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-18T00:45:42.059 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-18T00:45:42.060 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:45:42.060 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-18T00:45:42.062 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-18T00:45:42.062 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.063 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-18T00:45:42.063 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:42.063 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:42.064 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.29825 2026-02-18T00:45:42.065 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-18T00:45:42.065 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-18T00:45:42.065 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:124: setup: mkdir -p td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.066 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: get_asok_dir 2026-02-18T00:45:42.066 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:42.066 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:42.066 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:125: setup: mkdir -p /tmp/ceph-asok.29825 2026-02-18T00:45:42.068 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: ulimit -n 2026-02-18T00:45:42.069 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:126: setup: '[' 4096 -le 1024 ']' 2026-02-18T00:45:42.069 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:129: setup: '[' -z '' ']' 2026-02-18T00:45:42.069 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:130: setup: trap 'teardown td/mon-stretched-cluster-uneven-weight 1' TERM HUP INT 2026-02-18T00:45:42.069 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:32: TEST_stretched_cluster_uneven_weight: run_mon td/mon-stretched-cluster-uneven-weight a --public-addr 127.0.0.1:7139 2026-02-18T00:45:42.069 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.069 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-18T00:45:42.070 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=a 2026-02-18T00:45:42.070 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-18T00:45:42.070 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretched-cluster-uneven-weight/a 2026-02-18T00:45:42.070 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id a --mkfs --mon-data=td/mon-stretched-cluster-uneven-weight/a --run-dir=td/mon-stretched-cluster-uneven-weight --public-addr 127.0.0.1:7139 2026-02-18T00:45:42.093 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-18T00:45:42.093 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:45:42.093 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:45:42.093 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:45:42.093 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:42.093 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:42.093 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:45:42.094 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id a --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretched-cluster-uneven-weight/a '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretched-cluster-uneven-weight/log --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7139 2026-02-18T00:45:42.122 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-18T00:45:42.123 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a fsid 2026-02-18T00:45:42.123 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:45:42.123 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-18T00:45:42.123 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-18T00:45:42.123 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-18T00:45:42.124 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-18T00:45:42.124 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-18T00:45:42.124 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-18T00:45:42.125 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:45:42.125 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:42.125 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:42.125 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.29825/ceph-mon.a.asok 2026-02-18T00:45:42.125 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:45:42.125 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.29825/ceph-mon.a.asok config get fsid 2026-02-18T00:45:42.180 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon a mon_host 2026-02-18T00:45:42.180 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:45:42.180 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=a 2026-02-18T00:45:42.180 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-18T00:45:42.181 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-18T00:45:42.181 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.a 2026-02-18T00:45:42.181 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.a 2026-02-18T00:45:42.181 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.a ']' 2026-02-18T00:45:42.181 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:45:42.181 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:42.181 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:42.182 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.29825/ceph-mon.a.asok 2026-02-18T00:45:42.182 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:45:42.182 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.29825/ceph-mon.a.asok config get mon_host 2026-02-18T00:45:42.238 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:33: TEST_stretched_cluster_uneven_weight: wait_for_quorum 300 1 2026-02-18T00:45:42.239 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2026-02-18T00:45:42.239 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=1 2026-02-18T00:45:42.239 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2026-02-18T00:45:42.239 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 1 ]] 2026-02-18T00:45:42.239 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-18T00:45:42.239 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-18T00:45:42.240 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771375842 2026-02-18T00:45:42.240 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-18T00:45:42.240 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771375542 -lt 1771375842 ]] 2026-02-18T00:45:42.241 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 1' 2026-02-18T00:45:42.241 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2026-02-18T00:45:42.355 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-18T00:45:42.355 INFO:tasks.workunit.client.0.trial038.stderr:{"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"cc9f3802-e18f-4055-8c63-577ea50ede4c","modified":"2026-02-18T00:45:42.083359Z","created":"2026-02-18T00:45:42.083359Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:45:42.356 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":3,"quorum":[0],"quorum_names":["a"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":1,"fsid":"cc9f3802-e18f-4055-8c63-577ea50ede4c","modified":"2026-02-18T00:45:42.083359Z","created":"2026-02-18T00:45:42.083359Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:45:42.356 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 1' 2026-02-18T00:45:42.367 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-18T00:45:42.367 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-18T00:45:42.367 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-18T00:45:42.367 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-18T00:45:42.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-18T00:45:42.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:35: TEST_stretched_cluster_uneven_weight: run_mon td/mon-stretched-cluster-uneven-weight b --public-addr 127.0.0.1:7141 2026-02-18T00:45:42.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:42.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-18T00:45:42.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=b 2026-02-18T00:45:42.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-18T00:45:42.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretched-cluster-uneven-weight/b 2026-02-18T00:45:42.368 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id b --mkfs --mon-data=td/mon-stretched-cluster-uneven-weight/b --run-dir=td/mon-stretched-cluster-uneven-weight --public-addr 127.0.0.1:7141 2026-02-18T00:45:42.387 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-18T00:45:42.387 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:45:42.387 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:45:42.387 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:45:42.387 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:42.387 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:42.387 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:45:42.388 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id b --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretched-cluster-uneven-weight/b '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretched-cluster-uneven-weight/log --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7141 2026-02-18T00:45:42.414 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-18T00:45:42.415 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b fsid 2026-02-18T00:45:42.415 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:45:42.415 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-18T00:45:42.415 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-18T00:45:42.416 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-18T00:45:42.417 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-18T00:45:42.417 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-18T00:45:42.418 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-18T00:45:42.418 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:45:42.418 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:42.418 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:42.418 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.29825/ceph-mon.b.asok 2026-02-18T00:45:42.418 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:45:42.418 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.29825/ceph-mon.b.asok config get fsid 2026-02-18T00:45:42.473 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon b mon_host 2026-02-18T00:45:42.473 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:45:42.473 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=b 2026-02-18T00:45:42.474 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-18T00:45:42.474 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-18T00:45:42.474 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.b 2026-02-18T00:45:42.474 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.b 2026-02-18T00:45:42.474 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.b ']' 2026-02-18T00:45:42.474 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:45:42.474 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:42.474 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:42.475 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.29825/ceph-mon.b.asok 2026-02-18T00:45:42.475 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:45:42.476 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.29825/ceph-mon.b.asok config get mon_host 2026-02-18T00:45:42.529 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:36: TEST_stretched_cluster_uneven_weight: CEPH_ARGS='--fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141' 2026-02-18T00:45:42.530 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:37: TEST_stretched_cluster_uneven_weight: wait_for_quorum 300 2 2026-02-18T00:45:42.530 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2026-02-18T00:45:42.530 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=2 2026-02-18T00:45:42.530 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2026-02-18T00:45:42.530 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 2 ]] 2026-02-18T00:45:42.530 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-18T00:45:42.530 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-18T00:45:42.531 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771375842 2026-02-18T00:45:42.531 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-18T00:45:42.532 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771375542 -lt 1771375842 ]] 2026-02-18T00:45:42.532 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 2' 2026-02-18T00:45:42.532 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2026-02-18T00:45:48.647 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-18T00:45:48.647 INFO:tasks.workunit.client.0.trial038.stderr:{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":2,"fsid":"cc9f3802-e18f-4055-8c63-577ea50ede4c","modified":"2026-02-18T00:45:42.413099Z","created":"2026-02-18T00:45:42.083359Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:45:48.648 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":8,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":2,"fsid":"cc9f3802-e18f-4055-8c63-577ea50ede4c","modified":"2026-02-18T00:45:42.413099Z","created":"2026-02-18T00:45:42.083359Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:45:48.648 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 2' 2026-02-18T00:45:48.657 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-18T00:45:48.657 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-18T00:45:48.657 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-18T00:45:48.657 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-18T00:45:48.657 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-18T00:45:48.658 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:39: TEST_stretched_cluster_uneven_weight: run_mon td/mon-stretched-cluster-uneven-weight c --public-addr 127.0.0.1:7142 2026-02-18T00:45:48.658 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:48.658 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-18T00:45:48.658 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=c 2026-02-18T00:45:48.658 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-18T00:45:48.658 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretched-cluster-uneven-weight/c 2026-02-18T00:45:48.658 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id c --mkfs --mon-data=td/mon-stretched-cluster-uneven-weight/c --run-dir=td/mon-stretched-cluster-uneven-weight --public-addr 127.0.0.1:7142 2026-02-18T00:45:48.677 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-18T00:45:48.678 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:45:48.678 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:45:48.678 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:45:48.678 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:48.678 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:48.678 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:45:48.678 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id c --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretched-cluster-uneven-weight/c '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretched-cluster-uneven-weight/log --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7142 2026-02-18T00:45:48.706 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-18T00:45:48.707 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c fsid 2026-02-18T00:45:48.707 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:45:48.707 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-18T00:45:48.707 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-18T00:45:48.708 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-18T00:45:48.709 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-18T00:45:48.709 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-18T00:45:48.709 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-18T00:45:48.709 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:45:48.709 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:48.709 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:48.709 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.29825/ceph-mon.c.asok 2026-02-18T00:45:48.709 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:45:48.709 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.29825/ceph-mon.c.asok config get fsid 2026-02-18T00:45:48.764 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon c mon_host 2026-02-18T00:45:48.765 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:45:48.765 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=c 2026-02-18T00:45:48.765 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-18T00:45:48.765 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-18T00:45:48.766 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.c 2026-02-18T00:45:48.766 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.c 2026-02-18T00:45:48.766 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.c ']' 2026-02-18T00:45:48.766 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:45:48.767 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:48.767 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:48.767 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.29825/ceph-mon.c.asok 2026-02-18T00:45:48.767 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:45:48.767 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.29825/ceph-mon.c.asok config get mon_host 2026-02-18T00:45:48.820 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:40: TEST_stretched_cluster_uneven_weight: CEPH_ARGS='--fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142' 2026-02-18T00:45:48.821 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:41: TEST_stretched_cluster_uneven_weight: wait_for_quorum 300 3 2026-02-18T00:45:48.821 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2026-02-18T00:45:48.821 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=3 2026-02-18T00:45:48.821 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2026-02-18T00:45:48.821 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 3 ]] 2026-02-18T00:45:48.821 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-18T00:45:48.821 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-18T00:45:48.822 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771375848 2026-02-18T00:45:48.822 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-18T00:45:48.823 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771375548 -lt 1771375848 ]] 2026-02-18T00:45:48.823 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 3' 2026-02-18T00:45:48.823 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2026-02-18T00:45:54.936 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-18T00:45:54.937 INFO:tasks.workunit.client.0.trial038.stderr:{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":3,"fsid":"cc9f3802-e18f-4055-8c63-577ea50ede4c","modified":"2026-02-18T00:45:48.708157Z","created":"2026-02-18T00:45:42.083359Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:45:54.937 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 3' 2026-02-18T00:45:54.937 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":12,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":3,"fsid":"cc9f3802-e18f-4055-8c63-577ea50ede4c","modified":"2026-02-18T00:45:48.708157Z","created":"2026-02-18T00:45:42.083359Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:45:54.948 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-18T00:45:54.948 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-18T00:45:54.948 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-18T00:45:54.948 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-18T00:45:54.948 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-18T00:45:54.949 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:43: TEST_stretched_cluster_uneven_weight: run_mon td/mon-stretched-cluster-uneven-weight d --public-addr 127.0.0.1:7143 2026-02-18T00:45:54.949 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:45:54.949 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-18T00:45:54.949 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=d 2026-02-18T00:45:54.949 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-18T00:45:54.949 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretched-cluster-uneven-weight/d 2026-02-18T00:45:54.949 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id d --mkfs --mon-data=td/mon-stretched-cluster-uneven-weight/d --run-dir=td/mon-stretched-cluster-uneven-weight --public-addr 127.0.0.1:7143 2026-02-18T00:45:54.969 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-18T00:45:54.969 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:45:54.969 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:45:54.969 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:45:54.969 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:54.969 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:54.970 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:45:54.970 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id d --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretched-cluster-uneven-weight/d '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretched-cluster-uneven-weight/log --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7143 2026-02-18T00:45:54.996 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-18T00:45:54.997 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d fsid 2026-02-18T00:45:54.997 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:45:54.997 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-18T00:45:54.997 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-18T00:45:54.998 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-18T00:45:54.998 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-18T00:45:54.998 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-18T00:45:54.999 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-18T00:45:54.999 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:45:54.999 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:54.999 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:54.999 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.29825/ceph-mon.d.asok 2026-02-18T00:45:54.999 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:45:54.999 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.29825/ceph-mon.d.asok config get fsid 2026-02-18T00:45:55.055 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon d mon_host 2026-02-18T00:45:55.055 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:45:55.055 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=d 2026-02-18T00:45:55.055 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-18T00:45:55.055 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-18T00:45:55.056 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.d 2026-02-18T00:45:55.056 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.d 2026-02-18T00:45:55.056 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.d ']' 2026-02-18T00:45:55.056 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:45:55.057 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:45:55.057 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:45:55.057 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.29825/ceph-mon.d.asok 2026-02-18T00:45:55.057 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:45:55.057 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.29825/ceph-mon.d.asok config get mon_host 2026-02-18T00:45:55.111 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:44: TEST_stretched_cluster_uneven_weight: CEPH_ARGS='--fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143' 2026-02-18T00:45:55.112 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:45: TEST_stretched_cluster_uneven_weight: wait_for_quorum 300 4 2026-02-18T00:45:55.112 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2026-02-18T00:45:55.112 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=4 2026-02-18T00:45:55.112 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2026-02-18T00:45:55.112 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 4 ]] 2026-02-18T00:45:55.112 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-18T00:45:55.112 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-18T00:45:55.113 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771375855 2026-02-18T00:45:55.113 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-18T00:45:55.114 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771375555 -lt 1771375855 ]] 2026-02-18T00:45:55.114 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 4' 2026-02-18T00:45:55.114 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2026-02-18T00:46:02.641 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-18T00:46:02.642 INFO:tasks.workunit.client.0.trial038.stderr:{"election_epoch":16,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":4,"fsid":"cc9f3802-e18f-4055-8c63-577ea50ede4c","modified":"2026-02-18T00:45:54.998468Z","created":"2026-02-18T00:45:42.083359Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:46:02.642 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":16,"quorum":[0,1,2,3],"quorum_names":["a","b","c","d"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":4,"fsid":"cc9f3802-e18f-4055-8c63-577ea50ede4c","modified":"2026-02-18T00:45:54.998468Z","created":"2026-02-18T00:45:42.083359Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:46:02.643 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 4' 2026-02-18T00:46:02.652 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-18T00:46:02.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-18T00:46:02.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-18T00:46:02.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-18T00:46:02.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-18T00:46:02.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:47: TEST_stretched_cluster_uneven_weight: run_mon td/mon-stretched-cluster-uneven-weight e --public-addr 127.0.0.1:7144 2026-02-18T00:46:02.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:439: run_mon: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:46:02.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:440: run_mon: shift 2026-02-18T00:46:02.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:441: run_mon: local id=e 2026-02-18T00:46:02.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:442: run_mon: shift 2026-02-18T00:46:02.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:443: run_mon: local data=td/mon-stretched-cluster-uneven-weight/e 2026-02-18T00:46:02.653 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:446: run_mon: ceph-mon --id e --mkfs --mon-data=td/mon-stretched-cluster-uneven-weight/e --run-dir=td/mon-stretched-cluster-uneven-weight --public-addr 127.0.0.1:7144 2026-02-18T00:46:02.674 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: get_asok_path 2026-02-18T00:46:02.674 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:46:02.674 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:46:02.674 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:46:02.674 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:46:02.674 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:46:02.674 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:46:02.674 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:453: run_mon: ceph-mon --id e --osd-failsafe-full-ratio=.99 --mon-osd-full-ratio=.99 --mon-data-avail-crit=1 --mon-data-avail-warn=5 --paxos-propose-interval=0.1 --osd-crush-chooseleaf-type=0 --debug-mon 20 --debug-ms 20 --debug-paxos 20 --chdir= --mon-data=td/mon-stretched-cluster-uneven-weight/e '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --mon-cluster-log-file=td/mon-stretched-cluster-uneven-weight/log --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mon-allow-pool-delete --mon-allow-pool-size-one --osd-pool-default-pg-autoscale-mode off --mon-osd-backfillfull-ratio .99 --mon-warn-on-insecure-global-id-reclaim-allowed=false --public-addr 127.0.0.1:7144 2026-02-18T00:46:02.703 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: cat 2026-02-18T00:46:02.704 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e fsid 2026-02-18T00:46:02.704 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:46:02.704 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2026-02-18T00:46:02.704 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=fsid 2026-02-18T00:46:02.704 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .fsid 2026-02-18T00:46:02.706 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2026-02-18T00:46:02.706 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2026-02-18T00:46:02.706 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2026-02-18T00:46:02.706 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:46:02.706 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:46:02.706 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:46:02.706 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.29825/ceph-mon.e.asok 2026-02-18T00:46:02.706 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:46:02.706 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.29825/ceph-mon.e.asok config get fsid 2026-02-18T00:46:02.761 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:478: run_mon: get_config mon e mon_host 2026-02-18T00:46:02.761 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1116: get_config: local daemon=mon 2026-02-18T00:46:02.762 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1117: get_config: local id=e 2026-02-18T00:46:02.762 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1118: get_config: local config=mon_host 2026-02-18T00:46:02.763 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1123: get_config: jq -r .mon_host 2026-02-18T00:46:02.763 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: get_asok_path mon.e 2026-02-18T00:46:02.763 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name=mon.e 2026-02-18T00:46:02.763 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n mon.e ']' 2026-02-18T00:46:02.763 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: get_asok_dir 2026-02-18T00:46:02.763 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:46:02.763 INFO:tasks.workunit.client.0.trial038.stderr:////home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:46:02.764 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:109: get_asok_path: echo /tmp/ceph-asok.29825/ceph-mon.e.asok 2026-02-18T00:46:02.764 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: CEPH_ARGS= 2026-02-18T00:46:02.764 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1120: get_config: ceph --format json daemon /tmp/ceph-asok.29825/ceph-mon.e.asok config get mon_host 2026-02-18T00:46:02.818 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:48: TEST_stretched_cluster_uneven_weight: CEPH_ARGS='--fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:46:02.818 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:49: TEST_stretched_cluster_uneven_weight: wait_for_quorum 300 5 2026-02-18T00:46:02.818 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1046: wait_for_quorum: local timeout=300 2026-02-18T00:46:02.818 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1047: wait_for_quorum: local quorumsize=5 2026-02-18T00:46:02.818 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1049: wait_for_quorum: [[ -z 300 ]] 2026-02-18T00:46:02.819 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1053: wait_for_quorum: [[ -z 5 ]] 2026-02-18T00:46:02.819 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1058: wait_for_quorum: no_quorum=1 2026-02-18T00:46:02.819 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: date +%s 2026-02-18T00:46:02.819 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1059: wait_for_quorum: wait_until=1771375862 2026-02-18T00:46:02.819 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: date +%s 2026-02-18T00:46:02.820 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1060: wait_for_quorum: [[ 1771375562 -lt 1771375862 ]] 2026-02-18T00:46:02.820 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1061: wait_for_quorum: jqfilter='.quorum | length == 5' 2026-02-18T00:46:02.821 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: timeout 300 ceph quorum_status --format=json 2026-02-18T00:46:08.935 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1062: wait_for_quorum: jqinput=' 2026-02-18T00:46:08.935 INFO:tasks.workunit.client.0.trial038.stderr:{"election_epoch":22,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":5,"fsid":"cc9f3802-e18f-4055-8c63-577ea50ede4c","modified":"2026-02-18T00:46:02.705611Z","created":"2026-02-18T00:45:42.083359Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7144","nonce":0}]},"addr":"127.0.0.1:7144/0","public_addr":"127.0.0.1:7144/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:46:08.936 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: jq '.quorum | length == 5' 2026-02-18T00:46:08.936 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: echo '{"election_epoch":22,"quorum":[0,1,2,3,4],"quorum_names":["a","b","c","d","e"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4544132024016699391","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"]},"monmap":{"epoch":5,"fsid":"cc9f3802-e18f-4055-8c63-577ea50ede4c","modified":"2026-02-18T00:46:02.705611Z","created":"2026-02-18T00:45:42.083359Z","min_mon_release":20,"min_mon_release_name":"tentacle","election_strategy":1,"disallowed_leaders":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid","tentacle","nvmeof_beacon_diff"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7139","nonce":0}]},"addr":"127.0.0.1:7139/0","public_addr":"127.0.0.1:7139/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7141","nonce":0}]},"addr":"127.0.0.1:7141/0","public_addr":"127.0.0.1:7141/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7142","nonce":0}]},"addr":"127.0.0.1:7142/0","public_addr":"127.0.0.1:7142/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":3,"name":"d","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7143","nonce":0}]},"addr":"127.0.0.1:7143/0","public_addr":"127.0.0.1:7143/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":4,"name":"e","public_addrs":{"addrvec":[{"type":"v2","addr":"127.0.0.1:7144","nonce":0}]},"addr":"127.0.0.1:7144/0","public_addr":"127.0.0.1:7144/0","priority":0,"weight":0,"crush_location":"{}"}]}}' 2026-02-18T00:46:08.944 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1063: wait_for_quorum: res=true 2026-02-18T00:46:08.945 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1064: wait_for_quorum: [[ true == \t\r\u\e ]] 2026-02-18T00:46:08.945 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1065: wait_for_quorum: no_quorum=0 2026-02-18T00:46:08.945 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1066: wait_for_quorum: break 2026-02-18T00:46:08.945 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1069: wait_for_quorum: return 0 2026-02-18T00:46:08.945 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:51: TEST_stretched_cluster_uneven_weight: ceph mon set election_strategy connectivity 2026-02-18T00:46:09.069 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:52: TEST_stretched_cluster_uneven_weight: ceph mon add disallowed_leader e 2026-02-18T00:46:14.245 INFO:tasks.workunit.client.0.trial038.stderr:mon.e is already disallowed 2026-02-18T00:46:14.256 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:54: TEST_stretched_cluster_uneven_weight: run_mgr td/mon-stretched-cluster-uneven-weight x 2026-02-18T00:46:14.256 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:46:14.256 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-18T00:46:14.256 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=x 2026-02-18T00:46:14.257 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-18T00:46:14.257 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mon-stretched-cluster-uneven-weight/x 2026-02-18T00:46:14.257 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-18T00:46:14.371 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-18T00:46:14.371 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:46:14.371 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:46:14.371 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:46:14.371 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:46:14.371 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:46:14.371 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:46:14.372 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-18T00:46:14.372 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id x --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mon-stretched-cluster-uneven-weight/x '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-18T00:46:14.391 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:55: TEST_stretched_cluster_uneven_weight: run_mgr td/mon-stretched-cluster-uneven-weight y 2026-02-18T00:46:14.391 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:46:14.391 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-18T00:46:14.391 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=y 2026-02-18T00:46:14.391 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-18T00:46:14.391 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mon-stretched-cluster-uneven-weight/y 2026-02-18T00:46:14.391 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-18T00:46:14.504 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-18T00:46:14.505 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:46:14.505 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:46:14.505 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:46:14.505 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:46:14.505 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:46:14.505 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:46:14.506 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-18T00:46:14.506 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id y --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mon-stretched-cluster-uneven-weight/y '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-18T00:46:17.526 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:56: TEST_stretched_cluster_uneven_weight: run_mgr td/mon-stretched-cluster-uneven-weight z 2026-02-18T00:46:17.526 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:544: run_mgr: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:46:17.526 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:545: run_mgr: shift 2026-02-18T00:46:17.526 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:546: run_mgr: local id=z 2026-02-18T00:46:17.527 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:547: run_mgr: shift 2026-02-18T00:46:17.527 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:548: run_mgr: local data=td/mon-stretched-cluster-uneven-weight/z 2026-02-18T00:46:17.527 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:550: run_mgr: ceph config set mgr mgr_pool false --force 2026-02-18T00:46:17.732 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: get_asok_path 2026-02-18T00:46:17.732 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:46:17.732 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:46:17.732 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:46:17.733 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:46:17.733 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:46:17.733 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:46:17.733 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: realpath /home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-18T00:46:17.734 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:552: run_mgr: ceph-mgr --id z --osd-failsafe-full-ratio=.99 --debug-mgr 20 --debug-objecter 20 --debug-ms 20 --debug-paxos 20 --chdir= --mgr-data=td/mon-stretched-cluster-uneven-weight/z '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --run-dir=td/mon-stretched-cluster-uneven-weight '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --mgr-module-path=/home/ubuntu/cephtest/clone.client.0/src/pybind/mgr 2026-02-18T00:46:17.751 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:58: TEST_stretched_cluster_uneven_weight: expr 4 - 1 2026-02-18T00:46:17.752 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:58: TEST_stretched_cluster_uneven_weight: seq 0 3 2026-02-18T00:46:17.756 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:58: TEST_stretched_cluster_uneven_weight: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-18T00:46:17.756 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:60: TEST_stretched_cluster_uneven_weight: run_osd td/mon-stretched-cluster-uneven-weight 0 2026-02-18T00:46:17.756 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:46:17.756 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:46:17.756 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=0 2026-02-18T00:46:17.756 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:46:17.756 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretched-cluster-uneven-weight/0 2026-02-18T00:46:17.756 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:46:17.756 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:46:17.756 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:46:17.757 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:46:17.757 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretched-cluster-uneven-weight/0' 2026-02-18T00:46:17.757 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretched-cluster-uneven-weight/0/journal' 2026-02-18T00:46:17.757 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:46:17.757 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:46:17.757 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretched-cluster-uneven-weight' 2026-02-18T00:46:17.757 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:46:17.757 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:46:17.757 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:46:17.757 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:46:17.757 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:46:17.757 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:46:17.757 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:46:17.758 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:46:17.758 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:46:17.758 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:46:17.758 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:46:17.758 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretched-cluster-uneven-weight/$name.log' 2026-02-18T00:46:17.758 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' 2026-02-18T00:46:17.758 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:46:17.758 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:46:17.758 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:46:17.758 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:46:17.758 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:46:17.758 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:46:17.758 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretched-cluster-uneven-weight/0 2026-02-18T00:46:17.759 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:46:17.759 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=81a9ddd5-4848-4ebb-8e64-30ed982f7e72 2026-02-18T00:46:17.759 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd0 81a9ddd5-4848-4ebb-8e64-30ed982f7e72' 2026-02-18T00:46:17.760 INFO:tasks.workunit.client.0.trial038.stdout:add osd0 81a9ddd5-4848-4ebb-8e64-30ed982f7e72 2026-02-18T00:46:17.760 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:46:17.767 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDZC5Vp2o2dLRAAddMZbl7jMe3xAS4LodjFfg== 2026-02-18T00:46:17.767 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDZC5Vp2o2dLRAAddMZbl7jMe3xAS4LodjFfg=="}' 2026-02-18T00:46:17.767 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 81a9ddd5-4848-4ebb-8e64-30ed982f7e72 -i td/mon-stretched-cluster-uneven-weight/0/new.json 2026-02-18T00:46:17.987 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:46:17.998 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretched-cluster-uneven-weight/0/new.json 2026-02-18T00:46:17.999 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 0 --fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/0 --osd-journal=td/mon-stretched-cluster-uneven-weight/0/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDZC5Vp2o2dLRAAddMZbl7jMe3xAS4LodjFfg== --osd-uuid 81a9ddd5-4848-4ebb-8e64-30ed982f7e72 2026-02-18T00:46:18.015 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:18.010+0000 7f54fc053a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:18.017 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:18.014+0000 7f54fc053a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:18.018 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:18.014+0000 7f54fc053a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:18.019 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:18.014+0000 7f54fc053a40 -1 bdev(0x55f101bcc800 td/mon-stretched-cluster-uneven-weight/0/block) open stat got: (1) Operation not permitted 2026-02-18T00:46:18.019 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:18.014+0000 7f54fc053a40 -1 bluestore(td/mon-stretched-cluster-uneven-weight/0) _read_fsid unparsable uuid 2026-02-18T00:46:18.903 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretched-cluster-uneven-weight/0/keyring 2026-02-18T00:46:18.904 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:46:18.904 INFO:tasks.workunit.client.0.trial038.stdout:adding osd0 key to auth repository 2026-02-18T00:46:18.905 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd0 key to auth repository 2026-02-18T00:46:18.905 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretched-cluster-uneven-weight/0/keyring auth add osd.0 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:46:19.202 INFO:tasks.workunit.client.0.trial038.stdout:start osd.0 2026-02-18T00:46:19.203 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.0 2026-02-18T00:46:19.203 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 0 --fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/0 --osd-journal=td/mon-stretched-cluster-uneven-weight/0/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:46:19.203 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:46:19.204 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:46:19.204 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:46:19.225 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:19.222+0000 7fa74cd09a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:19.227 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:19.222+0000 7fa74cd09a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:19.228 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:19.222+0000 7fa74cd09a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:19.418 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 0 2026-02-18T00:46:19.419 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:46:19.419 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=0 2026-02-18T00:46:19.419 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:46:19.419 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:46:19.419 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:46:19.419 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:46:19.420 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:46:19.420 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:46:19.420 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-18T00:46:19.640 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:46:19.805 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:19.802+0000 7fa74cd09a40 -1 Falling back to public interface 2026-02-18T00:46:20.534 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:20.530+0000 7fa74cd09a40 -1 osd.0 0 log_to_monitors true 2026-02-18T00:46:20.641 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:46:20.641 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:46:20.642 INFO:tasks.workunit.client.0.trial038.stdout:1 2026-02-18T00:46:20.642 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-18T00:46:20.642 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:46:20.643 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-18T00:46:20.863 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:46:21.864 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:46:21.864 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:46:21.865 INFO:tasks.workunit.client.0.trial038.stdout:2 2026-02-18T00:46:21.865 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-18T00:46:21.866 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:46:21.866 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.0 up' 2026-02-18T00:46:22.082 INFO:tasks.workunit.client.0.trial038.stdout:osd.0 up in weight 1 up_from 5 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6802/785548886,v1:127.0.0.1:6803/785548886] [v2:127.0.0.1:6804/785548886,v1:127.0.0.1:6805/785548886] exists,up 81a9ddd5-4848-4ebb-8e64-30ed982f7e72 2026-02-18T00:46:22.086 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:46:22.086 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:46:22.086 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:46:22.086 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:58: TEST_stretched_cluster_uneven_weight: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-18T00:46:22.086 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:60: TEST_stretched_cluster_uneven_weight: run_osd td/mon-stretched-cluster-uneven-weight 1 2026-02-18T00:46:22.086 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:46:22.086 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:46:22.086 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=1 2026-02-18T00:46:22.086 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:46:22.086 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretched-cluster-uneven-weight/1 2026-02-18T00:46:22.087 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:46:22.087 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:46:22.087 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:46:22.087 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:46:22.087 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretched-cluster-uneven-weight/1' 2026-02-18T00:46:22.087 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretched-cluster-uneven-weight/1/journal' 2026-02-18T00:46:22.087 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:46:22.087 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:46:22.087 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretched-cluster-uneven-weight' 2026-02-18T00:46:22.087 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:46:22.088 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:46:22.088 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:46:22.088 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:46:22.088 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:46:22.088 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:46:22.088 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:46:22.088 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:46:22.088 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:46:22.088 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:46:22.088 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:46:22.088 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretched-cluster-uneven-weight/$name.log' 2026-02-18T00:46:22.088 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' 2026-02-18T00:46:22.088 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:46:22.089 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:46:22.089 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:46:22.089 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:46:22.089 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:46:22.089 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:46:22.089 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretched-cluster-uneven-weight/1 2026-02-18T00:46:22.089 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:46:22.090 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=4011e4b2-5183-4c41-83b8-5941b74bfd3a 2026-02-18T00:46:22.090 INFO:tasks.workunit.client.0.trial038.stdout:add osd1 4011e4b2-5183-4c41-83b8-5941b74bfd3a 2026-02-18T00:46:22.090 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd1 4011e4b2-5183-4c41-83b8-5941b74bfd3a' 2026-02-18T00:46:22.090 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:46:22.098 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDeC5Vp1ELBBRAAmLxdFoLXaDND/PZQ8JuAig== 2026-02-18T00:46:22.098 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDeC5Vp1ELBBRAAmLxdFoLXaDND/PZQ8JuAig=="}' 2026-02-18T00:46:22.099 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 4011e4b2-5183-4c41-83b8-5941b74bfd3a -i td/mon-stretched-cluster-uneven-weight/1/new.json 2026-02-18T00:46:22.318 INFO:tasks.workunit.client.0.trial038.stdout:1 2026-02-18T00:46:22.328 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretched-cluster-uneven-weight/1/new.json 2026-02-18T00:46:22.330 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 1 --fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/1 --osd-journal=td/mon-stretched-cluster-uneven-weight/1/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDeC5Vp1ELBBRAAmLxdFoLXaDND/PZQ8JuAig== --osd-uuid 4011e4b2-5183-4c41-83b8-5941b74bfd3a 2026-02-18T00:46:22.353 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:22.350+0000 7f305e267a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:22.354 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:22.350+0000 7f305e267a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:22.355 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:22.350+0000 7f305e267a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:22.356 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:22.354+0000 7f305e267a40 -1 bdev(0x562249c18800 td/mon-stretched-cluster-uneven-weight/1/block) open stat got: (1) Operation not permitted 2026-02-18T00:46:22.356 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:22.354+0000 7f305e267a40 -1 bluestore(td/mon-stretched-cluster-uneven-weight/1) _read_fsid unparsable uuid 2026-02-18T00:46:23.019 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretched-cluster-uneven-weight/1/keyring 2026-02-18T00:46:23.019 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:46:23.020 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd1 key to auth repository 2026-02-18T00:46:23.021 INFO:tasks.workunit.client.0.trial038.stdout:adding osd1 key to auth repository 2026-02-18T00:46:23.021 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretched-cluster-uneven-weight/1/keyring auth add osd.1 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:46:26.314 INFO:tasks.workunit.client.0.trial038.stdout:start osd.1 2026-02-18T00:46:26.315 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.1 2026-02-18T00:46:26.316 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 1 --fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/1 --osd-journal=td/mon-stretched-cluster-uneven-weight/1/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:46:26.316 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:46:26.316 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:46:26.316 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:46:26.336 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:26.334+0000 7f2367e50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:26.338 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:26.334+0000 7f2367e50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:26.339 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:26.334+0000 7f2367e50a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:26.526 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 1 2026-02-18T00:46:26.526 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:46:26.526 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=1 2026-02-18T00:46:26.526 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:46:26.527 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:46:26.527 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:46:26.527 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:46:26.528 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:46:26.528 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:46:26.528 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-18T00:46:26.736 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:46:26.749 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:26.746+0000 7f2367e50a40 -1 Falling back to public interface 2026-02-18T00:46:27.430 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:27.426+0000 7f2367e50a40 -1 osd.1 0 log_to_monitors true 2026-02-18T00:46:27.738 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:46:27.738 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:46:27.738 INFO:tasks.workunit.client.0.trial038.stdout:1 2026-02-18T00:46:27.739 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-18T00:46:27.739 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:46:27.739 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-18T00:46:27.953 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:46:28.955 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:46:28.955 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:46:28.955 INFO:tasks.workunit.client.0.trial038.stdout:2 2026-02-18T00:46:28.956 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-18T00:46:28.956 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:46:28.956 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.1 up' 2026-02-18T00:46:29.170 INFO:tasks.workunit.client.0.trial038.stdout:osd.1 up in weight 1 up_from 10 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6810/2262372571,v1:127.0.0.1:6811/2262372571] [v2:127.0.0.1:6812/2262372571,v1:127.0.0.1:6813/2262372571] exists,up 4011e4b2-5183-4c41-83b8-5941b74bfd3a 2026-02-18T00:46:29.174 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:46:29.174 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:46:29.174 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:46:29.174 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:58: TEST_stretched_cluster_uneven_weight: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-18T00:46:29.174 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:60: TEST_stretched_cluster_uneven_weight: run_osd td/mon-stretched-cluster-uneven-weight 2 2026-02-18T00:46:29.175 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:46:29.175 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:46:29.175 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=2 2026-02-18T00:46:29.175 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:46:29.175 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretched-cluster-uneven-weight/2 2026-02-18T00:46:29.175 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:46:29.175 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:46:29.175 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:46:29.175 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:46:29.175 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretched-cluster-uneven-weight/2' 2026-02-18T00:46:29.175 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretched-cluster-uneven-weight/2/journal' 2026-02-18T00:46:29.176 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:46:29.176 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:46:29.176 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretched-cluster-uneven-weight' 2026-02-18T00:46:29.176 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:46:29.176 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:46:29.176 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:46:29.176 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:46:29.176 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:46:29.176 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:46:29.176 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:46:29.176 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:46:29.176 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:46:29.176 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:46:29.177 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:46:29.177 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretched-cluster-uneven-weight/$name.log' 2026-02-18T00:46:29.177 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' 2026-02-18T00:46:29.177 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:46:29.177 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:46:29.177 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:46:29.177 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:46:29.177 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:46:29.177 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:46:29.177 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretched-cluster-uneven-weight/2 2026-02-18T00:46:29.178 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:46:29.178 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=38be315e-201b-4fb9-8973-5b13b3f4c4af 2026-02-18T00:46:29.178 INFO:tasks.workunit.client.0.trial038.stdout:add osd2 38be315e-201b-4fb9-8973-5b13b3f4c4af 2026-02-18T00:46:29.179 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd2 38be315e-201b-4fb9-8973-5b13b3f4c4af' 2026-02-18T00:46:29.179 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:46:29.186 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDlC5VpNNX6ChAAq1+KYGpl6Ws3WcOgH+CRQg== 2026-02-18T00:46:29.186 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDlC5VpNNX6ChAAq1+KYGpl6Ws3WcOgH+CRQg=="}' 2026-02-18T00:46:29.186 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 38be315e-201b-4fb9-8973-5b13b3f4c4af -i td/mon-stretched-cluster-uneven-weight/2/new.json 2026-02-18T00:46:29.406 INFO:tasks.workunit.client.0.trial038.stdout:2 2026-02-18T00:46:29.416 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretched-cluster-uneven-weight/2/new.json 2026-02-18T00:46:29.418 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 2 --fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/2 --osd-journal=td/mon-stretched-cluster-uneven-weight/2/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDlC5VpNNX6ChAAq1+KYGpl6Ws3WcOgH+CRQg== --osd-uuid 38be315e-201b-4fb9-8973-5b13b3f4c4af 2026-02-18T00:46:29.441 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:29.437+0000 7fbb912d6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:32.444 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:32.441+0000 7fbb912d6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:32.445 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:32.441+0000 7fbb912d6a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:32.446 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:32.441+0000 7fbb912d6a40 -1 bdev(0x55843d196800 td/mon-stretched-cluster-uneven-weight/2/block) open stat got: (1) Operation not permitted 2026-02-18T00:46:32.446 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:32.441+0000 7fbb912d6a40 -1 bluestore(td/mon-stretched-cluster-uneven-weight/2) _read_fsid unparsable uuid 2026-02-18T00:46:33.151 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretched-cluster-uneven-weight/2/keyring 2026-02-18T00:46:33.151 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:46:33.152 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd2 key to auth repository 2026-02-18T00:46:33.152 INFO:tasks.workunit.client.0.trial038.stdout:adding osd2 key to auth repository 2026-02-18T00:46:33.153 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretched-cluster-uneven-weight/2/keyring auth add osd.2 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:46:33.440 INFO:tasks.workunit.client.0.trial038.stdout:start osd.2 2026-02-18T00:46:33.441 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.2 2026-02-18T00:46:33.441 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 2 --fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/2 --osd-journal=td/mon-stretched-cluster-uneven-weight/2/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:46:33.441 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:46:33.441 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:46:33.441 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:46:33.456 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:33.449+0000 7f52d2524a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:33.457 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:33.453+0000 7f52d2524a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:33.458 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:33.453+0000 7f52d2524a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:33.648 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 2 2026-02-18T00:46:33.648 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:46:33.648 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=2 2026-02-18T00:46:33.649 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:46:33.649 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:46:33.649 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:46:33.649 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:46:33.650 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:46:33.650 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:46:33.650 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-18T00:46:33.867 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:46:33.893 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:33.889+0000 7f52d2524a40 -1 Falling back to public interface 2026-02-18T00:46:34.493 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:34.489+0000 7f52d2524a40 -1 osd.2 0 log_to_monitors true 2026-02-18T00:46:34.868 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:46:34.868 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:46:34.869 INFO:tasks.workunit.client.0.trial038.stdout:1 2026-02-18T00:46:34.869 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-18T00:46:34.870 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:46:34.870 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-18T00:46:35.085 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:46:36.086 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:46:36.086 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:46:36.087 INFO:tasks.workunit.client.0.trial038.stdout:2 2026-02-18T00:46:36.087 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-18T00:46:36.088 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:46:36.088 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.2 up' 2026-02-18T00:46:36.298 INFO:tasks.workunit.client.0.trial038.stdout:osd.2 up in weight 1 up_from 15 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6818/2220842828,v1:127.0.0.1:6819/2220842828] [v2:127.0.0.1:6820/2220842828,v1:127.0.0.1:6821/2220842828] exists,up 38be315e-201b-4fb9-8973-5b13b3f4c4af 2026-02-18T00:46:36.302 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:46:36.302 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:46:36.303 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:46:36.303 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:58: TEST_stretched_cluster_uneven_weight: for osd in $(seq 0 $(expr $OSDS - 1)) 2026-02-18T00:46:36.303 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:60: TEST_stretched_cluster_uneven_weight: run_osd td/mon-stretched-cluster-uneven-weight 3 2026-02-18T00:46:36.303 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:624: run_osd: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:46:36.303 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:625: run_osd: shift 2026-02-18T00:46:36.303 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:626: run_osd: local id=3 2026-02-18T00:46:36.303 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:627: run_osd: shift 2026-02-18T00:46:36.303 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:628: run_osd: local osd_data=td/mon-stretched-cluster-uneven-weight/3 2026-02-18T00:46:36.303 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:630: run_osd: local 'ceph_args=--fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144' 2026-02-18T00:46:36.303 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:631: run_osd: ceph_args+=' --osd-failsafe-full-ratio=.99' 2026-02-18T00:46:36.304 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:632: run_osd: ceph_args+=' --osd-journal-size=100' 2026-02-18T00:46:36.304 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:633: run_osd: ceph_args+=' --osd-scrub-load-threshold=2000' 2026-02-18T00:46:36.304 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:634: run_osd: ceph_args+=' --osd-data=td/mon-stretched-cluster-uneven-weight/3' 2026-02-18T00:46:36.304 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:635: run_osd: ceph_args+=' --osd-journal=td/mon-stretched-cluster-uneven-weight/3/journal' 2026-02-18T00:46:36.304 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:636: run_osd: ceph_args+=' --chdir=' 2026-02-18T00:46:36.304 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:637: run_osd: ceph_args+= 2026-02-18T00:46:36.304 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:638: run_osd: ceph_args+=' --run-dir=td/mon-stretched-cluster-uneven-weight' 2026-02-18T00:46:36.304 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: get_asok_path 2026-02-18T00:46:36.304 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:107: get_asok_path: local name= 2026-02-18T00:46:36.304 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:108: get_asok_path: '[' -n '' ']' 2026-02-18T00:46:36.304 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: get_asok_dir 2026-02-18T00:46:36.304 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:46:36.305 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:46:36.305 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:111: get_asok_path: echo '/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:46:36.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:639: run_osd: ceph_args+=' --admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' 2026-02-18T00:46:36.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:640: run_osd: ceph_args+=' --debug-osd=20' 2026-02-18T00:46:36.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:641: run_osd: ceph_args+=' --debug-ms=1' 2026-02-18T00:46:36.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:642: run_osd: ceph_args+=' --debug-monc=20' 2026-02-18T00:46:36.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:643: run_osd: ceph_args+=' --log-file=td/mon-stretched-cluster-uneven-weight/$name.log' 2026-02-18T00:46:36.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:644: run_osd: ceph_args+=' --pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' 2026-02-18T00:46:36.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:645: run_osd: ceph_args+=' --osd-max-object-name-len=460' 2026-02-18T00:46:36.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:646: run_osd: ceph_args+=' --osd-max-object-namespace-len=64' 2026-02-18T00:46:36.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:647: run_osd: ceph_args+=' --enable-experimental-unrecoverable-data-corrupting-features=*' 2026-02-18T00:46:36.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:648: run_osd: ceph_args+=' --osd-mclock-profile=high_recovery_ops' 2026-02-18T00:46:36.306 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:649: run_osd: ceph_args+=' ' 2026-02-18T00:46:36.306 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:650: run_osd: ceph_args+= 2026-02-18T00:46:36.306 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:651: run_osd: mkdir -p td/mon-stretched-cluster-uneven-weight/3 2026-02-18T00:46:36.306 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: uuidgen 2026-02-18T00:46:36.306 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:653: run_osd: local uuid=0eb3d151-77b9-40ce-b2a8-3c841bd2fd94 2026-02-18T00:46:36.307 INFO:tasks.workunit.client.0.trial038.stdout:add osd3 0eb3d151-77b9-40ce-b2a8-3c841bd2fd94 2026-02-18T00:46:36.307 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:654: run_osd: echo 'add osd3 0eb3d151-77b9-40ce-b2a8-3c841bd2fd94' 2026-02-18T00:46:36.307 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: ceph-authtool --gen-print-key 2026-02-18T00:46:36.316 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:655: run_osd: OSD_SECRET=AQDsC5Vpqi26EhAA47vU8wdrrEcXHQ5MtsHJ4A== 2026-02-18T00:46:36.316 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:656: run_osd: echo '{"cephx_secret": "AQDsC5Vpqi26EhAA47vU8wdrrEcXHQ5MtsHJ4A=="}' 2026-02-18T00:46:36.316 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:657: run_osd: ceph osd new 0eb3d151-77b9-40ce-b2a8-3c841bd2fd94 -i td/mon-stretched-cluster-uneven-weight/3/new.json 2026-02-18T00:46:36.534 INFO:tasks.workunit.client.0.trial038.stdout:3 2026-02-18T00:46:36.545 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:658: run_osd: rm td/mon-stretched-cluster-uneven-weight/3/new.json 2026-02-18T00:46:36.546 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:659: run_osd: ceph-osd -i 3 --fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/3 --osd-journal=td/mon-stretched-cluster-uneven-weight/3/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops --mkfs --key AQDsC5Vpqi26EhAA47vU8wdrrEcXHQ5MtsHJ4A== --osd-uuid 0eb3d151-77b9-40ce-b2a8-3c841bd2fd94 2026-02-18T00:46:36.561 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:36.557+0000 7fae2d4c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:36.562 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:36.557+0000 7fae2d4c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:36.563 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:36.557+0000 7fae2d4c5a40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:36.564 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:36.561+0000 7fae2d4c5a40 -1 bdev(0x55c7114cc800 td/mon-stretched-cluster-uneven-weight/3/block) open stat got: (1) Operation not permitted 2026-02-18T00:46:36.564 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:36.561+0000 7fae2d4c5a40 -1 bluestore(td/mon-stretched-cluster-uneven-weight/3) _read_fsid unparsable uuid 2026-02-18T00:46:37.302 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:661: run_osd: local key_fn=td/mon-stretched-cluster-uneven-weight/3/keyring 2026-02-18T00:46:37.302 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:662: run_osd: cat 2026-02-18T00:46:37.303 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:666: run_osd: echo adding osd3 key to auth repository 2026-02-18T00:46:37.304 INFO:tasks.workunit.client.0.trial038.stdout:adding osd3 key to auth repository 2026-02-18T00:46:37.304 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:667: run_osd: ceph -i td/mon-stretched-cluster-uneven-weight/3/keyring auth add osd.3 osd 'allow *' mon 'allow profile osd' mgr 'allow profile osd' 2026-02-18T00:46:37.600 INFO:tasks.workunit.client.0.trial038.stdout:start osd.3 2026-02-18T00:46:37.601 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:668: run_osd: echo start osd.3 2026-02-18T00:46:37.601 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:669: run_osd: ceph-osd -i 3 --fsid=cc9f3802-e18f-4055-8c63-577ea50ede4c --auth-supported=none --mon-host=127.0.0.1:7139,127.0.0.1:7141,127.0.0.1:7142,127.0.0.1:7143,127.0.0.1:7144 --osd-failsafe-full-ratio=.99 --osd-journal-size=100 --osd-scrub-load-threshold=2000 --osd-data=td/mon-stretched-cluster-uneven-weight/3 --osd-journal=td/mon-stretched-cluster-uneven-weight/3/journal --chdir= --run-dir=td/mon-stretched-cluster-uneven-weight '--admin-socket=/tmp/ceph-asok.29825/$cluster-$name.asok' --debug-osd=20 --debug-ms=1 --debug-monc=20 '--log-file=td/mon-stretched-cluster-uneven-weight/$name.log' '--pid-file=td/mon-stretched-cluster-uneven-weight/$name.pid' --osd-max-object-name-len=460 --osd-max-object-namespace-len=64 '--enable-experimental-unrecoverable-data-corrupting-features=*' --osd-mclock-profile=high_recovery_ops 2026-02-18T00:46:37.601 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: ceph osd dump --format=json 2026-02-18T00:46:37.601 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: jq '.flags_set[]' 2026-02-18T00:46:37.602 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:672: run_osd: grep -q '"noup"' 2026-02-18T00:46:37.621 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:37.617+0000 7f72f929da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:37.623 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:37.617+0000 7f72f929da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:37.624 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:37.621+0000 7f72f929da40 -1 WARNING: all dangerous and experimental features are enabled. 2026-02-18T00:46:37.809 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:675: run_osd: wait_for_osd up 3 2026-02-18T00:46:37.809 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:969: wait_for_osd: local state=up 2026-02-18T00:46:37.809 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:970: wait_for_osd: local id=3 2026-02-18T00:46:37.810 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:972: wait_for_osd: status=1 2026-02-18T00:46:37.810 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i=0 )) 2026-02-18T00:46:37.810 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:46:37.810 INFO:tasks.workunit.client.0.trial038.stdout:0 2026-02-18T00:46:37.811 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 0 2026-02-18T00:46:37.811 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:46:37.812 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-18T00:46:38.025 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:46:38.089 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:38.085+0000 7f72f929da40 -1 Falling back to public interface 2026-02-18T00:46:38.670 INFO:tasks.workunit.client.0.trial038.stderr:2026-02-18T00:46:38.665+0000 7f72f929da40 -1 osd.3 0 log_to_monitors true 2026-02-18T00:46:39.026 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:46:39.027 INFO:tasks.workunit.client.0.trial038.stdout:1 2026-02-18T00:46:39.027 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:46:39.027 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 1 2026-02-18T00:46:39.027 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:46:39.028 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-18T00:46:39.251 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:976: wait_for_osd: sleep 1 2026-02-18T00:46:40.252 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i++ )) 2026-02-18T00:46:40.252 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:973: wait_for_osd: (( i < 300 )) 2026-02-18T00:46:40.253 INFO:tasks.workunit.client.0.trial038.stdout:2 2026-02-18T00:46:40.253 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:974: wait_for_osd: echo 2 2026-02-18T00:46:40.253 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: ceph osd dump 2026-02-18T00:46:40.254 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:975: wait_for_osd: grep 'osd.3 up' 2026-02-18T00:46:40.464 INFO:tasks.workunit.client.0.trial038.stdout:osd.3 up in weight 1 up_from 20 up_thru 0 down_at 0 last_clean_interval [0,0) [v2:127.0.0.1:6826/133293143,v1:127.0.0.1:6827/133293143] [v2:127.0.0.1:6828/133293143,v1:127.0.0.1:6829/133293143] exists,up 0eb3d151-77b9-40ce-b2a8-3c841bd2fd94 2026-02-18T00:46:40.465 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:978: wait_for_osd: status=0 2026-02-18T00:46:40.465 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:979: wait_for_osd: break 2026-02-18T00:46:40.465 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:982: wait_for_osd: return 0 2026-02-18T00:46:40.465 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:63: TEST_stretched_cluster_uneven_weight: for zone in iris pze 2026-02-18T00:46:40.465 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:65: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket iris zone 2026-02-18T00:46:40.717 INFO:tasks.workunit.client.0.trial038.stderr:bucket 'iris' already exists 2026-02-18T00:46:40.727 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:66: TEST_stretched_cluster_uneven_weight: ceph osd crush move iris root=default 2026-02-18T00:46:40.999 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id -5 name 'iris' to location {root=default} in crush map 2026-02-18T00:46:41.009 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:63: TEST_stretched_cluster_uneven_weight: for zone in iris pze 2026-02-18T00:46:41.009 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:65: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket pze zone 2026-02-18T00:46:41.278 INFO:tasks.workunit.client.0.trial038.stderr:bucket 'pze' already exists 2026-02-18T00:46:41.289 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:66: TEST_stretched_cluster_uneven_weight: ceph osd crush move pze root=default 2026-02-18T00:46:41.588 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id -7 name 'pze' to location {root=default} in crush map 2026-02-18T00:46:41.599 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:69: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket node-2 host 2026-02-18T00:46:44.872 INFO:tasks.workunit.client.0.trial038.stderr:bucket 'node-2' already exists 2026-02-18T00:46:44.882 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:70: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket node-3 host 2026-02-18T00:46:45.154 INFO:tasks.workunit.client.0.trial038.stderr:bucket 'node-3' already exists 2026-02-18T00:46:45.164 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:71: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket node-4 host 2026-02-18T00:46:45.434 INFO:tasks.workunit.client.0.trial038.stderr:bucket 'node-4' already exists 2026-02-18T00:46:45.444 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:72: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket node-5 host 2026-02-18T00:46:45.716 INFO:tasks.workunit.client.0.trial038.stderr:bucket 'node-5' already exists 2026-02-18T00:46:45.726 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:74: TEST_stretched_cluster_uneven_weight: ceph osd crush move node-2 zone=iris 2026-02-18T00:46:45.997 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id -9 name 'node-2' to location {zone=iris} in crush map 2026-02-18T00:46:46.007 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:75: TEST_stretched_cluster_uneven_weight: ceph osd crush move node-3 zone=iris 2026-02-18T00:46:46.281 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id -10 name 'node-3' to location {zone=iris} in crush map 2026-02-18T00:46:46.293 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:76: TEST_stretched_cluster_uneven_weight: ceph osd crush move node-4 zone=pze 2026-02-18T00:46:46.553 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id -11 name 'node-4' to location {zone=pze} in crush map 2026-02-18T00:46:46.565 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:77: TEST_stretched_cluster_uneven_weight: ceph osd crush move node-5 zone=pze 2026-02-18T00:46:46.829 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id -12 name 'node-5' to location {zone=pze} in crush map 2026-02-18T00:46:46.839 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:79: TEST_stretched_cluster_uneven_weight: ceph osd crush move osd.0 host=node-2 2026-02-18T00:46:47.099 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id 0 name 'osd.0' to location {host=node-2} in crush map 2026-02-18T00:46:47.111 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:80: TEST_stretched_cluster_uneven_weight: ceph osd crush move osd.1 host=node-3 2026-02-18T00:46:47.384 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id 1 name 'osd.1' to location {host=node-3} in crush map 2026-02-18T00:46:47.395 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:81: TEST_stretched_cluster_uneven_weight: ceph osd crush move osd.2 host=node-4 2026-02-18T00:46:47.674 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id 2 name 'osd.2' to location {host=node-4} in crush map 2026-02-18T00:46:47.684 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:82: TEST_stretched_cluster_uneven_weight: ceph osd crush move osd.3 host=node-5 2026-02-18T00:46:47.959 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id 3 name 'osd.3' to location {host=node-5} in crush map 2026-02-18T00:46:47.969 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:84: TEST_stretched_cluster_uneven_weight: ceph mon set_location a zone=iris host=node-2 2026-02-18T00:46:48.276 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:85: TEST_stretched_cluster_uneven_weight: ceph mon set_location b zone=iris host=node-3 2026-02-18T00:46:48.587 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:86: TEST_stretched_cluster_uneven_weight: ceph mon set_location c zone=pze host=node-4 2026-02-18T00:47:12.923 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:87: TEST_stretched_cluster_uneven_weight: ceph mon set_location d zone=pze host=node-5 2026-02-18T00:47:18.204 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:89: TEST_stretched_cluster_uneven_weight: hostname -s 2026-02-18T00:47:18.205 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:89: TEST_stretched_cluster_uneven_weight: hostname=trial038 2026-02-18T00:47:18.205 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:90: TEST_stretched_cluster_uneven_weight: ceph osd crush remove trial038 2026-02-18T00:47:18.450 INFO:tasks.workunit.client.0.trial038.stderr:device 'trial038' does not appear in the crush map 2026-02-18T00:47:18.460 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:91: TEST_stretched_cluster_uneven_weight: ceph osd getcrushmap 2026-02-18T00:47:18.666 INFO:tasks.workunit.client.0.trial038.stderr:26 2026-02-18T00:47:18.675 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:92: TEST_stretched_cluster_uneven_weight: crushtool --decompile crushmap 2026-02-18T00:47:18.686 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:93: TEST_stretched_cluster_uneven_weight: sed 's/^# end crush map$//' crushmap.txt 2026-02-18T00:47:18.687 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:94: TEST_stretched_cluster_uneven_weight: cat 2026-02-18T00:47:18.688 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:110: TEST_stretched_cluster_uneven_weight: crushtool --compile crushmap_modified.txt -o crushmap.bin 2026-02-18T00:47:18.699 INFO:tasks.workunit.client.0.trial038.stderr:WARNING: min_size is no longer supported, ignoring 2026-02-18T00:47:18.699 INFO:tasks.workunit.client.0.trial038.stderr:WARNING: max_size is no longer supported, ignoring 2026-02-18T00:47:18.700 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:111: TEST_stretched_cluster_uneven_weight: ceph osd setcrushmap -i crushmap.bin 2026-02-18T00:47:19.022 INFO:tasks.workunit.client.0.trial038.stderr:28 2026-02-18T00:47:19.033 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:112: TEST_stretched_cluster_uneven_weight: local stretched_poolname=stretched_rbdpool 2026-02-18T00:47:19.033 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:113: TEST_stretched_cluster_uneven_weight: ceph osd pool create stretched_rbdpool 32 32 stretch_rule 2026-02-18T00:47:19.279 INFO:tasks.workunit.client.0.trial038.stderr:pool 'stretched_rbdpool' already exists 2026-02-18T00:47:19.289 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:114: TEST_stretched_cluster_uneven_weight: ceph osd pool set stretched_rbdpool size 4 2026-02-18T00:47:19.642 INFO:tasks.workunit.client.0.trial038.stderr:set pool 1 size to 4 2026-02-18T00:47:19.657 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:116: TEST_stretched_cluster_uneven_weight: ceph mon set_location e zone=arbiter host=node-1 2026-02-18T00:47:24.949 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:117: TEST_stretched_cluster_uneven_weight: ceph mon enable_stretch_mode e stretch_rule zone 2026-02-18T00:47:25.234 INFO:tasks.workunit.client.0.trial038.stderr:Second attempt of previously successful command failed with EINVAL: stretch mode is already engaged 2026-02-18T00:47:25.234 INFO:tasks.workunit.client.0.trial038.stderr:stretch mode is already engaged 2026-02-18T00:47:25.245 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:120: TEST_stretched_cluster_uneven_weight: ceph osd crush reweight osd.0 0.09000 2026-02-18T00:47:25.553 INFO:tasks.workunit.client.0.trial038.stderr:reweighted item id 0 name 'osd.0' to 0.09 in crush map 2026-02-18T00:47:25.567 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:121: TEST_stretched_cluster_uneven_weight: ceph osd crush reweight osd.1 0.09000 2026-02-18T00:47:28.880 INFO:tasks.workunit.client.0.trial038.stderr:reweighted item id 1 name 'osd.1' to 0.09 in crush map 2026-02-18T00:47:28.892 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:122: TEST_stretched_cluster_uneven_weight: ceph osd crush reweight osd.2 0.09000 2026-02-18T00:47:29.191 INFO:tasks.workunit.client.0.trial038.stderr:reweighted item id 2 name 'osd.2' to 0.09 in crush map 2026-02-18T00:47:29.203 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:123: TEST_stretched_cluster_uneven_weight: ceph osd crush reweight osd.3 0.09000 2026-02-18T00:47:29.500 INFO:tasks.workunit.client.0.trial038.stderr:reweighted item id 3 name 'osd.3' to 0.09 in crush map 2026-02-18T00:47:29.512 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:126: TEST_stretched_cluster_uneven_weight: ceph osd crush add-bucket sham zone 2026-02-18T00:47:29.760 INFO:tasks.workunit.client.0.trial038.stderr:bucket 'sham' already exists 2026-02-18T00:47:29.771 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:127: TEST_stretched_cluster_uneven_weight: ceph osd crush move sham root=default 2026-02-18T00:47:30.047 INFO:tasks.workunit.client.0.trial038.stderr:no need to move item id -3 name 'sham' to location {root=default} in crush map 2026-02-18T00:47:30.057 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:128: TEST_stretched_cluster_uneven_weight: wait_for_health INCORRECT_NUM_BUCKETS_STRETCH_MODE 2026-02-18T00:47:30.057 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1829: wait_for_health: local grepstr=INCORRECT_NUM_BUCKETS_STRETCH_MODE 2026-02-18T00:47:30.058 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: wait_for_health: get_timeout_delays 300 .1 2026-02-18T00:47:30.058 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-18T00:47:30.058 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-18T00:47:30.059 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-18T00:47:30.059 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-18T00:47:30.059 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-18T00:47:30.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: wait_for_health: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2026-02-18T00:47:30.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: wait_for_health: local -a delays 2026-02-18T00:47:30.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1831: wait_for_health: local -i loop=0 2026-02-18T00:47:30.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: ceph health detail 2026-02-18T00:47:30.281 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: grep INCORRECT_NUM_BUCKETS_STRETCH_MODE 2026-02-18T00:47:30.552 INFO:tasks.workunit.client.0.trial038.stdout:[WRN] INCORRECT_NUM_BUCKETS_STRETCH_MODE: Stretch mode buckets != 2 2026-02-18T00:47:30.553 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:130: TEST_stretched_cluster_uneven_weight: ceph osd crush rm sham 2026-02-18T00:47:30.774 INFO:tasks.workunit.client.0.trial038.stderr:device 'sham' does not appear in the crush map 2026-02-18T00:47:30.784 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:131: TEST_stretched_cluster_uneven_weight: wait_for_health_gone INCORRECT_NUM_BUCKETS_STRETCH_MODE 2026-02-18T00:47:30.785 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health_gone: local grepstr=INCORRECT_NUM_BUCKETS_STRETCH_MODE 2026-02-18T00:47:30.785 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1808: wait_for_health_gone: get_timeout_delays 300 .1 2026-02-18T00:47:30.786 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-18T00:47:30.786 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-18T00:47:30.786 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-18T00:47:30.787 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-18T00:47:30.787 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-18T00:47:31.002 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1808: wait_for_health_gone: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2026-02-18T00:47:31.002 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1808: wait_for_health_gone: local -a delays 2026-02-18T00:47:31.002 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1809: wait_for_health_gone: local -i loop=0 2026-02-18T00:47:31.002 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1811: wait_for_health_gone: ceph health detail 2026-02-18T00:47:31.002 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1811: wait_for_health_gone: grep INCORRECT_NUM_BUCKETS_STRETCH_MODE 2026-02-18T00:47:31.280 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:135: TEST_stretched_cluster_uneven_weight: ceph osd crush reweight osd.0 0.07000 2026-02-18T00:47:31.544 INFO:tasks.workunit.client.0.trial038.stderr:reweighted item id 0 name 'osd.0' to 0.07 in crush map 2026-02-18T00:47:31.557 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:137: TEST_stretched_cluster_uneven_weight: wait_for_health UNEVEN_WEIGHTS_STRETCH_MODE 2026-02-18T00:47:31.557 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1829: wait_for_health: local grepstr=UNEVEN_WEIGHTS_STRETCH_MODE 2026-02-18T00:47:31.557 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: wait_for_health: get_timeout_delays 300 .1 2026-02-18T00:47:31.558 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-18T00:47:31.558 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-18T00:47:31.558 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-18T00:47:31.559 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-18T00:47:31.559 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-18T00:47:31.785 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: wait_for_health: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2026-02-18T00:47:31.786 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1830: wait_for_health: local -a delays 2026-02-18T00:47:31.786 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1831: wait_for_health: local -i loop=0 2026-02-18T00:47:31.786 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: ceph health detail 2026-02-18T00:47:31.786 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1833: wait_for_health: grep UNEVEN_WEIGHTS_STRETCH_MODE 2026-02-18T00:47:35.068 INFO:tasks.workunit.client.0.trial038.stdout:[WRN] UNEVEN_WEIGHTS_STRETCH_MODE: Stretch mode buckets have different weights! 2026-02-18T00:47:35.068 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:139: TEST_stretched_cluster_uneven_weight: ceph osd crush reweight osd.0 0.09000 2026-02-18T00:47:35.384 INFO:tasks.workunit.client.0.trial038.stderr:reweighted item id 0 name 'osd.0' to 0.09 in crush map 2026-02-18T00:47:35.397 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:141: TEST_stretched_cluster_uneven_weight: wait_for_health_gone UNEVEN_WEIGHTS_STRETCH_MODE 2026-02-18T00:47:35.397 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1807: wait_for_health_gone: local grepstr=UNEVEN_WEIGHTS_STRETCH_MODE 2026-02-18T00:47:35.398 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1808: wait_for_health_gone: get_timeout_delays 300 .1 2026-02-18T00:47:35.398 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: shopt -q -o xtrace 2026-02-18T00:47:35.398 INFO:tasks.workunit.client.0.trial038.stderr:///home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: echo true 2026-02-18T00:47:35.399 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1593: get_timeout_delays: local trace=true 2026-02-18T00:47:35.399 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: true 2026-02-18T00:47:35.399 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1594: get_timeout_delays: shopt -u -o xtrace 2026-02-18T00:47:35.629 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1808: wait_for_health_gone: delays=('0.1' '0.2' '0.4' '0.8' '1.6' '3.2' '6.4' '12.8' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '15' '4.5') 2026-02-18T00:47:35.630 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1808: wait_for_health_gone: local -a delays 2026-02-18T00:47:35.630 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1809: wait_for_health_gone: local -i loop=0 2026-02-18T00:47:35.630 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1811: wait_for_health_gone: ceph health detail 2026-02-18T00:47:35.630 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:1811: wait_for_health_gone: grep UNEVEN_WEIGHTS_STRETCH_MODE 2026-02-18T00:47:35.915 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:143: TEST_stretched_cluster_uneven_weight: teardown td/mon-stretched-cluster-uneven-weight 2026-02-18T00:47:35.915 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:47:35.915 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-18T00:47:35.916 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretched-cluster-uneven-weight KILL 2026-02-18T00:47:35.916 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:47:35.916 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:47:35.916 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:47:35.916 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:47:35.916 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:47:36.228 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:47:36.228 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-18T00:47:36.229 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-18T00:47:36.230 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-18T00:47:36.232 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-18T00:47:36.232 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-18T00:47:36.232 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-18T00:47:36.233 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:47:36.233 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-18T00:47:36.234 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-18T00:47:36.234 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:47:36.235 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-18T00:47:36.236 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-18T00:47:36.236 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretched-cluster-uneven-weight 2026-02-18T00:47:36.285 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-18T00:47:36.285 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:47:36.285 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:47:36.285 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.29825 2026-02-18T00:47:36.286 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-18T00:47:36.286 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-18T00:47:36.287 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/mon-stretch/mon-stretch-uneven-crush-weights.sh:23: run: teardown td/mon-stretched-cluster-uneven-weight 2026-02-18T00:47:36.287 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:47:36.287 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs= 2026-02-18T00:47:36.287 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretched-cluster-uneven-weight KILL 2026-02-18T00:47:36.287 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:47:36.287 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:47:36.287 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:47:36.287 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:47:36.287 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:47:36.290 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:47:36.290 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-18T00:47:36.291 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-18T00:47:36.291 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-18T00:47:36.292 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-18T00:47:36.292 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-18T00:47:36.292 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-18T00:47:36.294 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:47:36.294 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-18T00:47:36.294 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-18T00:47:36.294 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:47:36.295 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-18T00:47:36.296 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o '' = 1 ']' 2026-02-18T00:47:36.296 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretched-cluster-uneven-weight 2026-02-18T00:47:36.298 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-18T00:47:36.298 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:47:36.298 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:47:36.298 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.29825 2026-02-18T00:47:36.299 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-18T00:47:36.300 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-18T00:47:36.300 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2401: main: code=0 2026-02-18T00:47:36.300 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2405: main: teardown td/mon-stretched-cluster-uneven-weight 0 2026-02-18T00:47:36.300 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:155: teardown: local dir=td/mon-stretched-cluster-uneven-weight 2026-02-18T00:47:36.300 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:156: teardown: local dumplogs=0 2026-02-18T00:47:36.300 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:157: teardown: kill_daemons td/mon-stretched-cluster-uneven-weight KILL 2026-02-18T00:47:36.300 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: shopt -q -o xtrace 2026-02-18T00:47:36.300 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: echo true 2026-02-18T00:47:36.300 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:336: kill_daemons: local trace=true 2026-02-18T00:47:36.300 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: true 2026-02-18T00:47:36.300 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:337: kill_daemons: shopt -u -o xtrace 2026-02-18T00:47:36.302 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:353: kill_daemons: return 0 2026-02-18T00:47:36.303 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: uname 2026-02-18T00:47:36.304 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:158: teardown: '[' Linux '!=' FreeBSD ']' 2026-02-18T00:47:36.304 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: stat -f -c %T . 2026-02-18T00:47:36.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:159: teardown: '[' ext2/ext3 == btrfs ']' 2026-02-18T00:47:36.305 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:162: teardown: local cores=no 2026-02-18T00:47:36.305 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: sysctl -n kernel.core_pattern 2026-02-18T00:47:36.306 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:163: teardown: local pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:47:36.306 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:165: teardown: '[' / = '|' ']' 2026-02-18T00:47:36.307 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: grep -q '^core\|core$' 2026-02-18T00:47:36.307 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: dirname /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2026-02-18T00:47:36.308 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:171: teardown: ls /home/ubuntu/cephtest/archive/coredump 2026-02-18T00:47:36.309 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:180: teardown: '[' no = yes -o 0 = 1 ']' 2026-02-18T00:47:36.309 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:189: teardown: rm -fr td/mon-stretched-cluster-uneven-weight 2026-02-18T00:47:36.310 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: get_asok_dir 2026-02-18T00:47:36.310 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:99: get_asok_dir: '[' -n '' ']' 2026-02-18T00:47:36.310 INFO:tasks.workunit.client.0.trial038.stderr://home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:102: get_asok_dir: echo /tmp/ceph-asok.29825 2026-02-18T00:47:36.310 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:190: teardown: rm -rf /tmp/ceph-asok.29825 2026-02-18T00:47:36.311 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:191: teardown: '[' no = yes ']' 2026-02-18T00:47:36.311 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:198: teardown: return 0 2026-02-18T00:47:36.312 INFO:tasks.workunit.client.0.trial038.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/ceph-helpers.sh:2406: main: return 0 2026-02-18T00:47:36.313 INFO:teuthology.orchestra.run:Running command with timeout 3600 2026-02-18T00:47:36.313 DEBUG:teuthology.orchestra.run.trial038:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2026-02-18T00:47:36.368 INFO:tasks.workunit:Stopping ['mon-stretch'] on client.0... 2026-02-18T00:47:36.368 DEBUG:teuthology.orchestra.run.trial038:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2026-02-18T00:47:36.725 DEBUG:teuthology.parallel:result is None 2026-02-18T00:47:36.726 DEBUG:teuthology.orchestra.run.trial038:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2026-02-18T00:47:36.735 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2026-02-18T00:47:36.735 DEBUG:teuthology.orchestra.run.trial038:> rmdir -- /home/ubuntu/cephtest/mnt.0 2026-02-18T00:47:36.779 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.0/client.0 2026-02-18T00:47:36.780 DEBUG:teuthology.run_tasks:Unwinding manager install 2026-02-18T00:47:36.797 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2026-02-18T00:47:36.797 DEBUG:teuthology.orchestra.run.trial038:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2026-02-18T00:47:36.929 DEBUG:teuthology.run_tasks:Unwinding manager clock 2026-02-18T00:47:36.940 INFO:teuthology.task.clock:Checking final clock skew... 2026-02-18T00:47:36.940 DEBUG:teuthology.orchestra.run.trial038:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2026-02-18T00:47:37.009 INFO:teuthology.orchestra.run.trial038.stdout: remote refid st t when poll reach delay offset jitter 2026-02-18T00:47:37.009 INFO:teuthology.orchestra.run.trial038.stdout:============================================================================== 2026-02-18T00:47:37.009 INFO:teuthology.orchestra.run.trial038.stdout:+blotch.image1te 75.127.194.146 2 u 4 64 77 20.114 -0.201 0.553 2026-02-18T00:47:37.009 INFO:teuthology.orchestra.run.trial038.stdout:+216.144.228.179 129.6.15.30 2 u 4 64 77 8.476 -0.149 0.493 2026-02-18T00:47:37.009 INFO:teuthology.orchestra.run.trial038.stdout:-s2-b.time.mci1. 42.20.202.230 2 u 4 64 77 29.785 -1.239 0.616 2026-02-18T00:47:37.009 INFO:teuthology.orchestra.run.trial038.stdout:*ntp2.zeus.frume 204.9.54.119 2 u 5 64 77 29.864 +0.320 1.602 2026-02-18T00:47:37.010 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2026-02-18T00:47:37.021 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2026-02-18T00:47:37.022 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2026-02-18T00:47:37.035 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2026-02-18T00:47:37.050 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2026-02-18T00:47:37.065 INFO:teuthology.task.internal:Duration was 405.136766 seconds 2026-02-18T00:47:37.065 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2026-02-18T00:47:37.085 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2026-02-18T00:47:37.085 DEBUG:teuthology.orchestra.run.trial038:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2026-02-18T00:47:37.111 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2026-02-18T00:47:37.111 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@trial038.front.sepia.ceph.com 2026-02-18T00:47:37.111 DEBUG:teuthology.orchestra.run.trial038:> grep -E --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | grep -E -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | grep -E -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2026-02-18T00:47:37.157 INFO:teuthology.task.internal.syslog:Gathering journactl... 2026-02-18T00:47:37.157 DEBUG:teuthology.orchestra.run.trial038:> sudo journalctl > /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-18T00:47:37.370 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2026-02-18T00:47:37.371 DEBUG:teuthology.orchestra.run.trial038:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --max-args=1 --max-procs=0 --verbose --no-run-if-empty -- gzip -5 --verbose -- 2026-02-18T00:47:37.379 INFO:teuthology.orchestra.run.trial038.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/misc.log 2026-02-18T00:47:37.381 INFO:teuthology.orchestra.run.trial038.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/journalctl.log 2026-02-18T00:47:37.381 INFO:teuthology.orchestra.run.trial038.stderr:/home/ubuntu/cephtest/archive/syslog/misc.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/misc.log.gz 2026-02-18T00:47:37.381 INFO:teuthology.orchestra.run.trial038.stderr:gzip -5 --verbose -- /home/ubuntu/cephtest/archive/syslog/kern.log 2026-02-18T00:47:37.381 INFO:teuthology.orchestra.run.trial038.stderr:/home/ubuntu/cephtest/archive/syslog/journalctl.log: /home/ubuntu/cephtest/archive/syslog/kern.log: 0.0% -- replaced with /home/ubuntu/cephtest/archive/syslog/kern.log.gz 2026-02-18T00:47:37.430 INFO:teuthology.orchestra.run.trial038.stderr: 81.7% -- replaced with /home/ubuntu/cephtest/archive/syslog/journalctl.log.gz 2026-02-18T00:47:37.431 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2026-02-18T00:47:37.447 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2026-02-18T00:47:37.447 DEBUG:teuthology.orchestra.run.trial038:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2026-02-18T00:47:37.483 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2026-02-18T00:47:37.499 DEBUG:teuthology.orchestra.run.trial038:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2026-02-18T00:47:37.535 INFO:teuthology.orchestra.run.trial038.stdout:kernel.core_pattern = core 2026-02-18T00:47:37.543 DEBUG:teuthology.orchestra.run.trial038:> test -e /home/ubuntu/cephtest/archive/coredump 2026-02-18T00:47:37.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2026-02-18T00:47:37.587 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2026-02-18T00:47:37.602 INFO:teuthology.task.internal:Transferring archived files... 2026-02-18T00:47:37.604 DEBUG:teuthology.misc:Transferring archived files from trial038:/home/ubuntu/cephtest/archive to /home/teuthworker/mnt/teuthology/yuriw-2026-02-17_21:14:21-rados-20.2.1_2.17.26-baseline_2-distro-default-trial/54187/remote/trial038 2026-02-18T00:47:37.605 DEBUG:teuthology.orchestra.run.trial038:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2026-02-18T00:47:37.659 INFO:teuthology.task.internal:Removing archive directory... 2026-02-18T00:47:37.659 DEBUG:teuthology.orchestra.run.trial038:> rm -rf -- /home/ubuntu/cephtest/archive 2026-02-18T00:47:37.691 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2026-02-18T00:47:37.708 INFO:teuthology.task.internal:Not uploading archives. 2026-02-18T00:47:37.709 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2026-02-18T00:47:37.725 INFO:teuthology.task.internal:Tidying up after the test... 2026-02-18T00:47:37.725 DEBUG:teuthology.orchestra.run.trial038:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2026-02-18T00:47:37.735 INFO:teuthology.orchestra.run.trial038.stdout: 1180220 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 18 00:47 /home/ubuntu/cephtest 2026-02-18T00:47:37.736 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2026-02-18T00:47:37.754 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2026-02-18T00:47:37.791 INFO:teuthology.run:Summary data: description: rados/standalone/{supported-random-distro$/{ubuntu_latest} workloads/mon-stretch} duration: 405.1367664337158 flavor: default owner: scheduled_yuriw@soko04.front.sepia.ceph.com success: true 2026-02-18T00:47:37.791 DEBUG:teuthology.report:Pushing job info to https://paddles-paddles.apps.pok.os.sepia.ceph.com 2026-02-18T00:47:37.906 INFO:teuthology.run:pass